A note from the maintainer: Follow-up questions (MaintNotes)

A note from the maintainer: Follow-up questions (MaintNotes)

[David Chanters]

Hi,

[Please retain a Cc back to me as I am not currently subscribed to the
mailing list.]

I've recently been intrigued with workflows, and have read quite a bit
about them, including various references on git-scm.com,
gitworkflows(7), and the email "A note from the maintainer" which I
have some questions on.  I'll paste random quotes from that and just
ask my question, I think, so apologies up front of it reads a little
disjointed.

I'd often wondered when I have read various posts of the git mailing
list on gmane, just how it is I am supposed to track:

dc/some-topic-feature

... Junio, are these topic branches ones you actively have somewhere
in your own private checkout?  Yes, I appreciate that when I read a
given post to the mailing list, you or other people will sometimes
make reference to these topic branches, but what do I do if I am
interested in finding out about one of them?  Indeed, perhaps even
before getting to that question, how do you go about creating and
maintaining these topic branches -- are you making heavy use of "git
am"?

I ask because of the following snippet from "MaintNotes":

    The two branches "master" and "maint" are never rewound, and
    "next" usually will not be either (this automatically means the
    topics that have been merged into "next" are usually not
    rebased, and you can find the tip of topic branches you are
    interested in from the output of "git log next"). You should be
    able to safely track them.

I am not sure if there's any real use-case for this, but I will ask
anyway:  is the above saying that I am able to *checkout* one of these
topic-branches just from their presence in "next" alone?  I appreciate
that the point is somewhat moot since the topic branch has already
been merged into "next", but I can surely see this as a really useful
way for people to manage topic-branches in a shared environment:
people can simply pick a topic branch out from the integrated one --
in this case "next".  Or is this idea a complete waste of time?

To continue:

    The "pu" (proposed updates) branch bundles all the remainder of
    topic branches.  The "pu" branch, and topic branches that are
    only in "pu", are subject to rebasing in general.  By the above
    definition of how "next" works, you can tell that this branch
    will contain quite experimental and obviously broken stuff.

I'm obviously missing something here -- but why is rebasing these
existing topic branches (I assume on top of "pu") more useful than
just merging them into "pu" -- like you do with "next"?

    When a topic that was in "pu" proves to be in testable shape, it
    graduates to "next".  I do this with:

            git checkout next
            git merge that-topic-branch

    Sometimes, an idea that looked promising turns out to be not so
    good and the topic can be dropped from "pu" in such a case.

Ah -- so if I have this straight in my head -- you continually form
the local topic-branch on its own branch, and then just merge it into
"next" only when you know that topic branch is satisfactory?  That
being the case -- again, I assume the use of "git am" for the topic
branch?  If regular readers of the git mailing list wish to track this
topic branch, can they do so from you only until it's merged into
"next"?

And a related question:  If you decide a given topic in pu is declared
to "be dropped", is this done by rebasing (as you mentioned earlier)
so as to remove any trace of the topic branch ever having been in
"pu", or am I reading too much into "dropping" here?  :)

I hope these aren't too idiotic.

Thanks all in advance.

David

Re: A note from the maintainer: Follow-up questions (MaintNotes)

[Andreas Ericsson]

David Chanters wrote:
> Hi,
> 
> [Please retain a Cc back to me as I am not currently subscribed to the
> mailing list.]
> 
> I've recently been intrigued with workflows, and have read quite a bit
> about them, including various references on git-scm.com,
> gitworkflows(7), and the email "A note from the maintainer" which I
> have some questions on.  I'll paste random quotes from that and just
> ask my question, I think, so apologies up front of it reads a little
> disjointed.
> 
> I'd often wondered when I have read various posts of the git mailing
> list on gmane, just how it is I am supposed to track:
> 
> dc/some-topic-feature
> 
> ... Junio, are these topic branches ones you actively have somewhere
> in your own private checkout?  Yes, I appreciate that when I read a
> given post to the mailing list, you or other people will sometimes
> make reference to these topic branches, but what do I do if I am
> interested in finding out about one of them?  Indeed, perhaps even
> before getting to that question, how do you go about creating and
> maintaining these topic branches -- are you making heavy use of "git
> am"?
> 
> I ask because of the following snippet from "MaintNotes":
> 
>     The two branches "master" and "maint" are never rewound, and
>     "next" usually will not be either (this automatically means the
>     topics that have been merged into "next" are usually not
>     rebased, and you can find the tip of topic branches you are
>     interested in from the output of "git log next"). You should be
>     able to safely track them.
> 
> I am not sure if there's any real use-case for this, but I will ask
> anyway:  is the above saying that I am able to *checkout* one of these
> topic-branches just from their presence in "next" alone?

Yes. "git log --grep=dc/some-topic-feature next" would point you to
the merge commit where it gets merged to 'next'. Then you can simply
do "git checkout -b dc/some-topic-feature (the-located-commit)^2" to
create the branch "dc/some-topic-feature" as it was when Junio merged
it. This relies on Junio not tampering with the commit messages git
creates when merging, but since there's no real need for that anyway,
it's a fairly safe practice.

> 
> To continue:
> 
>     The "pu" (proposed updates) branch bundles all the remainder of
>     topic branches.  The "pu" branch, and topic branches that are
>     only in "pu", are subject to rebasing in general.  By the above
>     definition of how "next" works, you can tell that this branch
>     will contain quite experimental and obviously broken stuff.
> 
> I'm obviously missing something here -- but why is rebasing these
> existing topic branches (I assume on top of "pu") more useful than
> just merging them into "pu" -- like you do with "next"?
> 

Because topics in 'pu' can be dropped on the floor, and are worked
on quite a lot more than the ones in next. Undoing a merge is quite
a lot of work as opposed to just rebuilding the history without that
merge. It's also a lot nicer to have a cleaner history in 'next',
since that makes it easier to merge things to 'master' in such a way
that bisection works nicely.

>     When a topic that was in "pu" proves to be in testable shape, it
>     graduates to "next".  I do this with:
> 
>             git checkout next
>             git merge that-topic-branch
> 
>     Sometimes, an idea that looked promising turns out to be not so
>     good and the topic can be dropped from "pu" in such a case.
> 
> Ah -- so if I have this straight in my head -- you continually form
> the local topic-branch on its own branch, and then just merge it into
> "next" only when you know that topic branch is satisfactory?

I *think* the topic branches live until they're merged to master, or
until they're dropped (although I could well imagine them staying
behind quite some time after being dropped). I haven't heard Junio
talk about this afair, but that's how I would do it anyways.

>  That
> being the case -- again, I assume the use of "git am" for the topic
> branch?

I should think so, although once a patch is applied in its final
form it'll no longer be necessary to use "git am". It'll be easier
to just use rebase on the topic branch(es), or cherry-pick individual
commits from them in order to get only certain benefits of them.
This isn't *very* usual, but it does happen from time to time that
topic branches refactor something first and then adds a feature on
top of that.

>  If regular readers of the git mailing list wish to track this
> topic branch, can they do so from you only until it's merged into
> "next"?
> 

Topics that make it to master can be tracked indefinitely. 'next'
is never merged directly to master, since the topics brewing on
next get different amounts of testing and feedback. Junio just
merges the topics to master as they go through all the review and
testing they're thought to need. Try

  git log --grep="Merge branch '" master

and you'll see what I mean, or have a look using gitk.

> And a related question:  If you decide a given topic in pu is declared
> to "be dropped", is this done by rebasing (as you mentioned earlier)
> so as to remove any trace of the topic branch ever having been in
> "pu", or am I reading too much into "dropping" here?  :)
> 

AFAIU, 'pu' is dropped in its entirety and re-built from the top of 'next'
by the rather simple expedient of "git reset --hard next" and then merging,
one by one, all branches that aren't already merged to maint, master or
next. Git can list such branches so it's no great chore. Naturally, Junio
knows his way fairly well around git and has scripts to do much of this
work for him.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: A note from the maintainer: Follow-up questions (MaintNotes)

[Junio C Hamano]

David Chanters <david.chanters@googlemail.com> writes:

> I'd often wondered when I have read various posts of the git mailing
> list on gmane, just how it is I am supposed to track:
>
> dc/some-topic-feature
>
> ... Junio, are these topic branches ones you actively have somewhere
> in your own private checkout?  Yes, I appreciate that when I read a
> given post to the mailing list, you or other people will sometimes
> make reference to these topic branches, but what do I do if I am
> interested in finding out about one of them?

    $ git log --oneline --first-parent origin/master..origin/pu

would be a handy way to view where the tip of each branch is.

    a71f64a Merge branch 'pk/import-dirs' into pu
    ce6cd39 Merge branch 'jh/cvs-helper' into pu
    ...
    2178d02 Merge branch 'jc/log-tz' into pu
    ...
    927d129 Merge branch 'lt/approxidate' into jch
    35ada54 Merge branch 'tr/reset-checkout-patch' into jch
    d82f86c Merge branch 'db/vcs-helper' (early part) into jch

So if you for example happen to be interested in jc/log-tz topic,
you would do something like:

    $ git checkout -b jc/log-tz 2178d02^2
    $ git log -p master..

to check out, and view what changes the topic introduces. 

Some hawk-eyed people may have already noticed this, but I recently
updated the script I used to maintain the "What's cooking" messages, and
the entries come with when and at what commit each part of the series has
been merged to 'next'.  For example, the lt/approxidate topic reads like
this:

  * lt/approxidate (2009-08-30) 6 commits
    (merged to 'next' on 2009-08-30 at e016e3d)
   + fix approxidate parsing of relative months and years
   + tests: add date printing and parsing tests
   + refactor test-date interface
   + Add date formatting and parsing functions relative to a given time
    (merged to 'next' on 2009-08-26 at 62853f9)
   + Further 'approxidate' improvements
   + Improve on 'approxidate'

Time flows from bottom to top in this output, so this tells us that the
first two patches in the series has been in 'next' for five days or so,
and the tests and a fix in 4 follow-up patches came later, merged to
'next' yesterday.  You can use

    $ git checkout -b lt/approxidate e016e3d^2

to get at its tip.

> ..., how do you go about creating and
> maintaining these topic branches -- are you making heavy use of "git
> am"

Save patches from the list in mbox, review them in the mbox while fixing
trivial breakages, and finally:

    $ git checkout -b ai/topic-name master
    $ git am -s that.mbox

where "ai" is typically the author's initial, and topic-name names the
topic just like you would name a function.  A topic typically forks from
the tip of master if it is a new feature, or a much older commit in maint
if it is a fix (and in such a case, topic-name typically begins with
a string "maint-").

> I ask because of the following snippet from "MaintNotes":
>
>     The two branches "master" and "maint" are never rewound, and
>     "next" usually will not be either (this automatically means the
>     topics that have been merged into "next" are usually not
>     rebased, and you can find the tip of topic branches you are
>     interested in from the output of "git log next"). You should be
>     able to safely track them.
>
> I am not sure if there's any real use-case for this, but I will ask
> anyway:  is the above saying that I am able to *checkout* one of these
> topic-branches just from their presence in "next" alone?  I appreciate
> that the point is somewhat moot since the topic branch has already
> been merged into "next", but I can surely see this as a really useful
> way for people to manage topic-branches in a shared environment:
> people can simply pick a topic branch out from the integrated one --
> in this case "next".

Surely, see above.  And by checking out the topic alone, you can test and
enhance it in isolation.

If you come up with a follow-up patch based on one particular topic, in
other words, building on ai/topic-name created like the above example, as
opposed to building on 'next', it would be easier for me to integrate it,
too, because the way I accept a follow-up patch to a particular topic is
by doing this:

    $ git checkout ai/topic-name
    $ git am -s followup-mail.mbox

The result will be tested in isolation and if it is good it would be
merged to 'next' again.

    $ git merge ai/topic-name

> I'm obviously missing something here -- but why is rebasing these
> existing topic branches (I assume on top of "pu") more useful than
> just merging them into "pu" -- like you do with "next"?

Somebody may send a few patches [PATCH 1/3] thru [PATCH 3/3] whose
quality is sub-par but tries to tackle a good problem.  I'd queue it in
'pu'.

    $ git checkout -b dc/cool-feature
    $ git am -s david-chanters.mbox
    $ git checkout pu
    $ git merge dc/cool-feature

In a few days, people would notice that the series has a lot of room for
improvement, and offer suggestions.  You would send replacement patches
based on the review.  Perhaps you squashed the first two patches from the
original into one, and added another patch for test suite, and the new
series is marked as [PATCH v2 1/3] thru [PATCH v2 3/3].

In general, the early attempts of a topic that have never been merged to
'next' are not worth keeping in the public history.  The original author
is free to keep them in his private tree, but there is no point in forever
keeping earlier mistakes, bugs, typos and implementation based on a flawed
design, all of which later were corrected in _my_ history.

So I would replace the whole topic with the new series.

    $ git checkout dc/cool-feature
    $ git reset --hard HEAD~3
    $ git am -s david-chanters-v2.mbox

The whole point of replacing the contents of this topic was to get rid of
the mistakes in your earlier round, so it does not make much sense to
merge this to 'pu' without rewinding 'pu' first.

After the topic is merged to 'next', obviously I cannot sanely rewind and
rebuild it.  Everything goes incremental from then on (see the earlier
example of applying followup-mail.mbox).

> If regular readers of the git mailing list wish to track this topic
> branch, can they do so from you only until it's merged into "next"?

Sorry, but I am not sure if I understand the question.

> And a related question:  If you decide a given topic in pu is declared
> to "be dropped", is this done by rebasing (as you mentioned earlier)
> so as to remove any trace of the topic branch ever having been in
> "pu", or am I reading too much into "dropping" here?  :)

Essentially, I keep a list of branches that are in 'next' and another list
of branches that are in 'pu'.  At the end of each integration cycle, I do
a rough equivalent of:

    $ git checkout next
    $ git merge ..a good topic that is not fully in next..
    $ git merge ..another good topic that is not fully in next..
    $ make test ;# make sure next is ok
    $ git branch -f pu ;# get rid of everything in pu
    $ git checkout pu
    $ for branch in ..the list of branches to be in pu..
      do
          git merge $branch || break
      done

Discarding a topic from 'pu' simply means I do not merge the topic in the
last loop.

Re: A note from the maintainer: Follow-up questions (MaintNotes)

[Jeff King]

On Mon, Aug 31, 2009 at 06:38:37PM -0700, Junio C Hamano wrote:

>     a71f64a Merge branch 'pk/import-dirs' into pu
>     ce6cd39 Merge branch 'jh/cvs-helper' into pu
>     ...
>     2178d02 Merge branch 'jc/log-tz' into pu
>     ...
>     927d129 Merge branch 'lt/approxidate' into jch
>     35ada54 Merge branch 'tr/reset-checkout-patch' into jch
>     d82f86c Merge branch 'db/vcs-helper' (early part) into jch
> 
> So if you for example happen to be interested in jc/log-tz topic,
> you would do something like:
> 
>     $ git checkout -b jc/log-tz 2178d02^2
>     $ git log -p master..
> 
> to check out, and view what changes the topic introduces.

Another alternative: Thomas Rast wrote a handy script called
'git-resurrect' which does this for you. It's carried in the contrib/
section right now. I used it just the other day to pull out the
lt/approxidate topic branch, which I then built some follow-up patches
for. Usage is something like:

  # contrib scripts aren't installed by "make install",
  # so put it somewhere in your PATH
  $ cp contrib/git-resurrect.sh ~/bin/git-resurrect

  $ git resurrect -m lt/approxidate
  ** Candidates for lt/approxidate **
  931e8e2 [22 hours ago] fix approxidate parsing of relative months and years
  ** Restoring lt/approxidate to 931e8e2 fix approxidate parsing of relative months and years

-Peff

Re: A note from the maintainer: Follow-up questions (MaintNotes)

[David Chanters]

2009/9/1 Junio C Hamano <gitster@pobox.com>:
>    $ git log --oneline --first-parent origin/master..origin/pu
>
> would be a handy way to view where the tip of each branch is.

Yes it is - thanks for that!  I presume that (in other workflows --
not necessarily git,git's) that using git-resurrect.sh would be
preferable to the git-log suggestion above if the topic branch wanting
to be "resurrected" had several merge points?

> So if you for example happen to be interested in jc/log-tz topic,
> you would do something like:
>
>    $ git checkout -b jc/log-tz 2178d02^2
>    $ git log -p master..
>
> to check out, and view what changes the topic introduces.

This is really useful - thank you - it's solving a missing piece of a
puzzle for me.  :)

> where "ai" is typically the author's initial, and topic-name names the
> topic just like you would name a function.  A topic typically forks from
> the tip of master if it is a new feature, or a much older commit in maint
> if it is a fix (and in such a case, topic-name typically begins with
> a string "maint-").

Makes sense - and on that note - in our current workflow of using Git,
we have a feature branch, call it "featureA" which is forked from
"master" (where our stable code lives) -- but obviously over time, if
bugfixes happen and get released, what do we do about then ensuring
that featureA also benefits from these bug-fixes?  Since invariably
people will want to develop using the bug-fixes, but "featureA" long
since branched from "master" at a point in the past, before the
bug-fixes?

What do you do, about this when handling topic branches merged into
next, or doesn't it really matter by that point?

[...snip really useful explanation...]

I can't thank you enough, Junio for this -- you've effectively ironed
out a workflow here I think I can now go away and start using -
thanks.  :)

David

Re: A note from the maintainer: Follow-up questions (MaintNotes)

[Junio C Hamano]

David Chanters <david.chanters@googlemail.com> writes:

> 2009/9/1 Junio C Hamano <gitster@pobox.com>:
>>    $ git log --oneline --first-parent origin/master..origin/pu
>>
>> would be a handy way to view where the tip of each branch is.
>
> Yes it is - thanks for that!  I presume that (in other workflows --
> not necessarily git,git's) that using git-resurrect.sh would be
> preferable to the git-log suggestion above if the topic branch wanting
> to be "resurrected" had several merge points?

AFAIK the script is an (attempt of) automating the inspecting of the above
output and manually choosing which one to use.  As automation, it may be
handy, and as automation, it may have bugs ;-).

> Makes sense - and on that note - in our current workflow of using Git,
> we have a feature branch, call it "featureA" which is forked from
> "master" (where our stable code lives) -- but obviously over time, if
> bugfixes happen and get released, what do we do about then ensuring
> that featureA also benefits from these bug-fixes?  Since invariably
> people will want to develop using the bug-fixes, but "featureA" long
> since branched from "master" at a point in the past, before the
> bug-fixes?

That is one of the most common misconception people new to the topic
branch workflow have [*1*].

In general, you do not want to merge fixes or other changes from
integration branches back into a topic branch such as featureA.  The
purpose of a topic branch is to brew the topic and the topic alone in
isolation to perfection.  Once you start merging 'maint' or 'master' into
'featureA', the output from "git log featureA" ceases to be "the work I
did to polish the feature A".  The branch becomes "this contains the work
related to feature A, but overwhelmingly consists of other unrelated
random stuff".  It would be useless as a topic branch, because you cannot
tell which ones are the changes to achieve the goal of the topic anymore.

When you test featureA, you would test the topic branch itself, and while
doing so, you will of course see bugs that have been fixed since you
forked from the mainline.  Do not get distracted by them; you did not fork
the featureA topic in order to fix them in the first place!

Of course, when you test the whole system including the addition of
featureA, you would want to make sure that fixes and enhancements that
happened on the mainline since you forked the topic branch work well with
what you did on the topic.

For that kind of integration testing, either use a temporary branch to
test trial integration:


    $ git checkout -b test master ;# start with everything else they did
    $ git merge featureA ;# ... and try my topic with that
    $ test..test..test

and once you are done, you can get rid of the test branch

    $ git checkout master
    $ git branch -D test

You can alternatively have a dedicated branch for early integration
testing.  That is what 'next' branch is used for in git.git project.

    $ git checkout next
    $ git merge featureA ;# ... merge the update
    $ test..test..test
    ... yeek, my changes were buggy ...
    $ git reset --hard HEAD^ ;# get rid of the change
    $ git checkout featureA
    $ fix..fix..fix
    $ git checkout next
    $ git merge featureA ;# ... try again
    $ test..test..test
    ... this time it is good and I am happy ...


[Footnote]

*1* I think I have had a chance to answer this question in a lot more
detail at least twice in the past couple of weeks on this list, and both
times it was very frustrating for me, not because I did not want to answer
them, but because I've written it in length in my upcoming book but
pointing to it would not help people who asked the question because it is
in Japanese.  I've been writing on a handful of topics I thought were
worthy of general consumption on my blog while writing the book, but this
is one of the topics that I haven't "backported" there yet X-<.