keeping track of where a patch begins

keeping track of where a patch begins

[E R]

Hi,

We've started to use git at work. Developers create branches for their
patches (which we call "tickets" because they are related to our
ticketing system), and those branches are picked up by an integration
team and merged together to form a release. Hopefully this is not too
unconventional.

Ideally a developer will start their ticket branch from a previous
release. However, occasionally a developer working on multiple tickets
will forget to switch back to a release node when creating a new
ticket branch. Then code from the first ticket inadvertently gets
added to the second ticket, and this is a problem if integration
decides to include the second ticket in the release but not the first.

What solutions have you come up with to either to catch or prevent
this from happening? It is possible to determine what node a branch
started from?

It seems that somehow the node that the patch begins at has to be
either identified, marked or remembered, and it might have to done
outside of git. Then the integration team or other tools can validate
the starting node to ensure that it complies with the build process.

Thanks,
ER

Re: keeping track of where a patch begins

[Nicolas Pitre]

On Wed, 21 Oct 2009, E R wrote:

> What solutions have you come up with to either to catch or prevent
> this from happening? It is possible to determine what node a branch
> started from?

This can be determined by looking at the gitk output.

Also 'git merge-base' can give you that node, given the main branch and 
the topic branch.  See documentation about git-merge-base.

Then if you need to move a branch to another starting node, then 'git 
rebase' is what you need (again the git-rebase documentation is pretty 
detailed).


Nicolas

Re: keeping track of where a patch begins

[Junio C Hamano]

Nicolas Pitre <nico@fluxnic.net> writes:

> On Wed, 21 Oct 2009, E R wrote:
>
>> What solutions have you come up with to either to catch or prevent
>> this from happening? It is possible to determine what node a branch
>> started from?
>
> This can be determined by looking at the gitk output.
>
> Also 'git merge-base' can give you that node, given the main branch and 
> the topic branch.  See documentation about git-merge-base.
>
> Then if you need to move a branch to another starting node, then 'git 
> rebase' is what you need (again the git-rebase documentation is pretty 
> detailed).

That is a correct way to diagnose the mistake and recover from it, but
unfortunately it is a rather weak tool to identify the mistake in the
first place.

A branch in git, as Randal often used to say on #git, is an illusion---it
points only at the top and does not identify the bottom.

But it does _not_ have to stay that way at the Porcelain level.

Here is a rough sketch of one possible solution.  It is not fully thought
out; the basic idea is probably sound but I did not try to exhaustively
cover changes to various tools that are necessary to maintain the
invariants this scheme requires.

 (0) Define a way to identify the bottom of a branch.  One way to do this
     is by an extra ref (e.g. refs/branchpoints/frotz).  Then the commits
     between refs/branchpoints/frotz..refs/heads/frotz identifies the
     commits on the branch.  None of the additional restrictions below
     applies when the branch does not have such bottom defined (i.e.
     created by the current git without this extension).

 (1) At branch creation, the branchpoint is noted.  E.g.

     $ git branch frotz master~4

     would internally become

     $ git update-ref refs/heads/frotz master~4
     $ git update-ref refs/branchpoints/frotz master~4

     You would also need to cover "checkout -b".

 (2) You can grow the branch naturally with "commit", "am" and "merge".
     The bottom of the branch does not have to move with these operations.

 (3) Operations that alter histories, e.g. "commit --amend", "rebase",
     "reset", while on a branch that records its bottom need to be taught
     to pay attention to not break its bottom.  Paying attention needs to
     take different forms depending on the operation; some probably will
     forbid the operation while others would automatically adjust the
     bottom.

     Examples (not exhaustive):

 (3-a) "branch -f frotz $commit"

     This moves the tip of the branch.  Unless $commit is already some
     part of the existing frotz branch, we should probably forbid it for
     simplicity, when a bottom is defined for the branch.

     We could later loosen the rule so that $commit is only required to be
     a descendant of existing bottom of the branch to support a workflow
     like this:

     $ git checkout -b frotz master~4 ;# records branchpoint
     $ edit; git add; git commit; ... ;# builds history
     $ git checkout HEAD^             ;# go back somewhere on frotz
     $ edit; git add; git commit; ... ;# builds an alternate history
     $ git show-branch HEAD frotz     ;# check progress
     $ git diff frotz HEAD            ;# is this one better?
     $ git branch -f frotz            ;# I prefer this new one better

 (3-b) "reset $commit" (with or without --hard/--soft/--mixed)

     This is similar to (3-a) above; $commit has to be a descendant of
     existing bottom.

 (3-c) "commit --amend"

     $ git checkout -b frotz master~4 ;# records branchpoint
     $ git commit --amend             ;# rewrite the bottom???

     would probably be a mistake, as the end result would make the frotz
     branch forked from master~5 with the first commit on the branch a
     fix-up to what is already in the master branch.

     However, this is a valid way to work:

     $ git checkout -b frotz master~4 ;# records branchpoint
     $ edit; git add; git commit      ;# builds history
     $ git commit --amend             ;# fix the tip

     and it does not have to do anything to the bottom.

 (3-d) "rebase"

     $ git checkout -b frotz master~4 ;# records branchpoint
     $ edit; git add; git commit; ... ;# builds history
     $ git rebase --onto master       ;# transplants the branch

     would make the "onto" commit the new bottom.  Another interesting
     thing to note is that we do not have to compute which commits to
     transplant with merge-base with the "onto" commit, because we know
     the bottom commit of the branch.

 (4) Operations that browse histories, e.g. "log", "show-branch", while on
     a branch that records its bottom can be taught to pay attention to
     the bottom.  For example, it is conceivable that

     $ git log
     $ git log -- Documentation/

     without an explicit branch name that fell back to the default HEAD
     while on branch "frotz" might be better run with an implicit bottom
     ^refs/branchpoint/frotz.

We probably could kill the other bird in the nearby thread that wants to
add a description to a branch, if this scheme is fully implemented (no, I
am not going to start coding right away, as this message is just a sketch
of what we _could_ do), As we will fully know in what operations we need
to update the branchpoint ref, we could make the refs/branchpoints/frotz
an annotated tag, and store the description for the branch in that tag.
Whenever we need to adjust the branchpoint, we update it while carrying
the branch description message over to the new tag object.

Re: keeping track of where a patch begins

[Nicolas Pitre]

On Wed, 21 Oct 2009, Junio C Hamano wrote:

> Nicolas Pitre <nico@fluxnic.net> writes:
> 
> > On Wed, 21 Oct 2009, E R wrote:
> >
> >> What solutions have you come up with to either to catch or prevent
> >> this from happening? It is possible to determine what node a branch
> >> started from?
> >
> > This can be determined by looking at the gitk output.
> >
> > Also 'git merge-base' can give you that node, given the main branch and 
> > the topic branch.  See documentation about git-merge-base.
> >
> > Then if you need to move a branch to another starting node, then 'git 
> > rebase' is what you need (again the git-rebase documentation is pretty 
> > detailed).
> 
> That is a correct way to diagnose the mistake and recover from it, but
> unfortunately it is a rather weak tool to identify the mistake in the
> first place.

Well... The "mistake" is probably going to be different depending on the 
work flow used.  I don't think there is a generic definition of such 
mistakes.

In this case, simply having

	if [ $(git merge-base $expected_branch_point $branch) != \
	     $(git rev-parse $expected_branch_point) ]; then
		(complain/refuse the merge of $branch)
	fi

should be quite sufficient as an enforcing proper branch policy.  Of 
course the $expected_branch_point is something that is determined 
outside of Git.

> A branch in git, as Randal often used to say on #git, is an illusion---it
> points only at the top and does not identify the bottom.
> 
> But it does _not_ have to stay that way at the Porcelain level.
> 
> Here is a rough sketch of one possible solution.  It is not fully thought
> out; the basic idea is probably sound but I did not try to exhaustively
> cover changes to various tools that are necessary to maintain the
> invariants this scheme requires.

I never came across a situation where such an elaborated scheme was 
needed to actually record and maintain that information, or could be 
really useful.  And some branches might be built on top of a sub-branch 
already, making the real branch's bottom the sub-branch's instead in a 
given context.  It all depends on the work flow and the convention used 
for a project.  And the tool has no way to figure that out (is this the 
real branch bottom or should it be one or more level down?), etc.

> We probably could kill the other bird in the nearby thread that wants to
> add a description to a branch, if this scheme is fully implemented

Well, I think we gain in flexibility by keeping those things separate 
though.  Blending data structures together is not always a good thing.

We have reflog data separate from the refs themselves, so I think that 
having .git/desc/refs/* containing simple text files would be good 
enough and simple to implement/use.


Nicolas

Re: keeping track of where a patch begins

[Thomas Rast]

Junio C Hamano wrote:
> 
> A branch in git, as Randal often used to say on #git, is an illusion---it
> points only at the top and does not identify the bottom.
> 
> But it does _not_ have to stay that way at the Porcelain level.
> 
> Here is a rough sketch of one possible solution.  It is not fully thought
> out; the basic idea is probably sound but I did not try to exhaustively
> cover changes to various tools that are necessary to maintain the
> invariants this scheme requires.
> 
>  (0) Define a way to identify the bottom of a branch.  One way to do this
>      is by an extra ref (e.g. refs/branchpoints/frotz).  Then the commits
>      between refs/branchpoints/frotz..refs/heads/frotz identifies the
>      commits on the branch.  None of the additional restrictions below
>      applies when the branch does not have such bottom defined (i.e.
>      created by the current git without this extension).
> 
>  (1) At branch creation, the branchpoint is noted. [...]
> 
>  (2) You can grow the branch naturally with "commit", "am" and "merge".
>      The bottom of the branch does not have to move with these operations.
> 
>  (3) Operations that alter histories, e.g. "commit --amend", "rebase",
>      "reset", while on a branch that records its bottom need to be taught
>      to pay attention to not break its bottom. [...]
> 
>  (4) Operations that browse histories, e.g. "log", "show-branch", while on
>      a branch that records its bottom can be taught to pay attention to
>      the bottom. [...]

I think this not only changes the model of branches, but also commits,
to some extent.  Currently, commit have no intrinsic branch
membership; if you say

  git branch foo bar

you cannot distinguish whether the commits on 'bar' were created on
'foo' or on 'bar'.  (By git's means; of course the decision would
favour 'master' if I had used that instead.)

Technically your proposal does not change this fact very much; it is
still possible to create "clones" of branches that are
indistinguishable.  However, to the *user* I think we would create a
notion that "a commit belongs to one specific branch", in that, during
the course of normal operations, a commit will end up on exactly one

  git rev-list --first-parent base..branch

range.

(Not sure if I consider this as an argument in favour or against yet,
but I wanted to point it out anyway.)

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: keeping track of where a patch begins

[Jeff King]

On Wed, Oct 21, 2009 at 01:03:55PM -0700, Junio C Hamano wrote:

>  (0) Define a way to identify the bottom of a branch.  One way to do this
>      is by an extra ref (e.g. refs/branchpoints/frotz).  Then the commits
>      between refs/branchpoints/frotz..refs/heads/frotz identifies the
>      commits on the branch.  None of the additional restrictions below
>      applies when the branch does not have such bottom defined (i.e.
>      created by the current git without this extension).

Hmm. This feels like redundant information to me. It has always been
git's strategy to record the history graph, and to use merge bases as
the "bottom" of branches, rather than keeping an artificial "started
here" commit. So I am trying to see the advantages of recording a static
bottom versus doing a merge-base calculation later. Some things I can
think of:

  - a bottom implies a specific commit, whereas a merge-base is always
    with respect to anothe tip. So to have a default "bottom" calculated
    by merge-base, you need a default "upstream". Which we do have, but
    of course it is subject to being rewound.

  - your merge-base will move when you merge. But arguably, that is a
    good thing. If you are talking about "git log" only looking at the
    commits on this branch (as you do later in the quoted email), I
    would expect to see only stuff that happened since upstream last
    merged. Although to be honest, I am not sure such a limit is all
    that useful. We already have "git log upstream..branch".

So I am not really clear on what you are trying to accomplish by
recording such a bottom. Your steps (0) through (3) seem to be leading
up to this use case:

>  (4) Operations that browse histories, e.g. "log", "show-branch", while on
>      a branch that records its bottom can be taught to pay attention to
>      the bottom.  For example, it is conceivable that
> 
>      $ git log
>      $ git log -- Documentation/
> 
>      without an explicit branch name that fell back to the default HEAD
>      while on branch "frotz" might be better run with an implicit bottom
>      ^refs/branchpoint/frotz.

If that is all you want, can't we just default to something like:

  $ git log $(git for-each-ref --format='%(upstream)' $(git symbolic-ref HEAD)))..

Of course it would be much easier to type as "git log @{upstream}.." :)

-Peff

Re: keeping track of where a patch begins

[Pascal Obry]

Le 22/10/2009 10:27, Thomas Rast a écrit :
> I think this not only changes the model of branches, but also commits,
> to some extent.  Currently, commit have no intrinsic branch
> membership; if you say
>
>    git branch foo bar
>
> you cannot distinguish whether the commits on 'bar' were created on
> 'foo' or on 'bar'.  (By git's means; of course the decision would
> favour 'master' if I had used that instead.)

I have been looking for a way to know that. I've even post a question 
about this on this mailing-list long time ago IIRC.

To me there is case where it is important to know which are the commits 
done on a topic branch for example. When working on multiple topic it is 
difficult to remember which commits have been done on this specific 
branch. This is needed to rebase onto:

    $ git rebase --onto somebranch <topic_base> <topic_head>

A common idiom, but one as to think hard (& right) to properly get the 
topic_base today.

Just my 2 cents!

Pascal.

-- 

--|------------------------------------------------------
--| Pascal Obry                           Team-Ada Member
--| 45, rue Gabriel Peri - 78114 Magny Les Hameaux FRANCE
--|------------------------------------------------------
--|    http://www.obry.net  -  http://v2p.fr.eu.org
--| "The best way to travel is by means of imagination"
--|
--| gpg --keyserver keys.gnupg.net --recv-key F949BD3B

Re: keeping track of where a patch begins

[Thomas Rast]

Pascal Obry wrote:
> 
> Le 22/10/2009 10:27, Thomas Rast a écrit :
> > I think this not only changes the model of branches, but also commits,
> > to some extent.  Currently, commit have no intrinsic branch
> > membership
[...]
> To me there is case where it is important to know which are the commits 
> done on a topic branch for example. When working on multiple topic it is 
> difficult to remember which commits have been done on this specific 
> branch. This is needed to rebase onto:
> 
>     $ git rebase --onto somebranch <topic_base> <topic_head>
> 
> A common idiom, but one as to think hard (& right) to properly get the 
> topic_base today.

But how frequently do your topics start on other topics?  Otherwise
they will start on an integration or maint branch, and in the git.git
model where each integration branch is contained in the next "less
stable" one, that means you can just specify 'pu' or equivalent.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch