What is the whole process of cooking git as a maintainer?

What is the whole process of cooking git as a maintainer?

[Ping Yin]

I have scanned the manual and found a 'Linux subsystem maintainer'
example. However, that's a little brief and i wonder a more real
example. I think an example of how to cook git as a maintainer is
better.

Junio , could you please share the whole process of cooking among pu,
next and master? Or, can i find this in the mail list archive?

For example:
In which branch the test is performed?
How to rewind pu?
How to merge a subset of changes from pu to next, and from next to master?


-- 
Ping Yin

Re: What is the whole process of cooking git as a maintainer?

[Ping Yin]

On Nov 21, 2007 9:11 PM, Ping Yin <pkufranky@gmail.com> wrote:
> I have scanned the manual and found a 'Linux subsystem maintainer'
> example. However, that's a little brief and i wonder a more real
> example. I think an example of how to cook git as a maintainer is
> better.
>
> Junio , could you please share the whole process of cooking among pu,
> next and master? Or, can i find this in the mail list archive?
>
> For example:
> In which branch the test is performed?
> How to rewind pu?
> How to merge a subset of changes from pu to next, and from next to master?
>

One more question:
I see so many merges in pu branch, but where these merges go when a
feature is moved to the master branch?
>
> --
> Ping Yin
>



-- 
Ping Yin

Re: What is the whole process of cooking git as a maintainer?

[Theodore Tso]

On Wed, Nov 21, 2007 at 09:17:59PM +0800, Ping Yin wrote:
> On Nov 21, 2007 9:11 PM, Ping Yin <pkufranky@gmail.com> wrote:
> > I have scanned the manual and found a 'Linux subsystem maintainer'
> > example. However, that's a little brief and i wonder a more real
> > example. I think an example of how to cook git as a maintainer is
> > better.
> >
> > Junio , could you please share the whole process of cooking among pu,
> > next and master? Or, can i find this in the mail list archive?
> >
> > For example:
> > In which branch the test is performed?
> > How to rewind pu?
> > How to merge a subset of changes from pu to next, and from next to master?
> >
> 
> One more question:
> I see so many merges in pu branch, but where these merges go when a
> feature is moved to the master branch?

There is a discussion of some of these issues in the mail thread
beginning here:

http://kerneltrap.org/mailarchive/git/2007/10/23/350943

The git setup is actually pretty complex, and all of the details of
how to maintain the pu and next branches are pretty complicated ---
and if you don't have a large development community, it may not be
worth the overhead, and it may scare people off as being too complicated.

There are multiple ways to maintain a project in git, and it's usually
better to start simple --- with a development and maintenance branch.
If you need more, you can add more later, but I wouldn't recommend
starting off with a very complex system right off the bat.

	     	    	 	 	      - Ted

Re: What is the whole process of cooking git as a maintainer?

[Johannes Schindelin]

Hi,

On Wed, 21 Nov 2007, Ping Yin wrote:

> Junio , could you please share the whole process of cooking among pu, 
> next and master? Or, can i find this in the mail list archive?

The easiest way is to read the notes from the maintainer, and then study 
the scripts in the 'todo' branch, and guess...

Hth,
Dscho

Addendum to "MaintNotes"

[Junio C Hamano]

The maintainer's git time is spent on three activities.

 - Communication (60%)

   Mailing list discussions on general design, fielding user
   questions, diagnosing bug reports; reviewing, commenting on,
   suggesting alternatives to, and rejecting patches.

 - Integration (30%)

   Applying new patches from the contributors while spotting and
   correcting minor mistakes, shuffling the integration and
   testing branches, pushing the results out, cutting the
   release, and making announcements.

 - Own development (10%)

   Scratching my own itch and sending proposed patch series out.

The policy on Integration is informally mentioned in "A Note
from the maintainer" message, which is periodically posted to
this mailing list after each feature release is made.

The policy.

 - Feature releases are numbered as vX.Y.Z and are meant to
   contain bugfixes and enhancements in any area, including
   functionality, performance and usability, without regression.

 - Maintenance releases are numbered as vX.Y.Z.W and are meant
   to contain only bugfixes for the corresponding vX.Y.Z feature
   release and earlier maintenance releases vX.Y.Z.V (V < W).

 - 'master' branch is used to prepare for the next feature
   release. In other words, at some point, the tip of 'master'
   branch is tagged with vX.Y.Z.

 - 'maint' branch is used to prepare for the next maintenance
   release.  After the feature release vX.Y.Z is made, the tip
   of 'maint' branch is set to that release, and bugfixes will
   accumulate on the branch, and at some point, the tip of the
   branch is tagged with vX.Y.Z.1, vX.Y.Z.2, and so on.

 - 'next' branch is used to publish changes (both enhancements
   and fixes) that (1) have worthwhile goal, (2) are in a fairly
   good shape suitable for everyday use, (3) but have not yet
   demonstrated to be regression free.  New changes are tested
   in 'next' before merged to 'master'.

 - 'pu' branch is used to publish other proposed changes that do
   not yet pass the criteria set for 'next'.

 - The tips of 'master', 'maint' and 'next' branches will always
   fast forward, to allow people to build their own
   customization on top of them.

 - Usually 'master' contains all of 'maint', 'next' contains all
   of 'master' and 'pu' contains all of 'next'.

 - The tip of 'master' is meant to be more stable than any
   tagged releases, and the users are encouraged to follow it.

 - The 'next' branch is where new action takes place, and the
   users are encouraged to test it so that regressions and bugs
   are found before new topics are merged to 'master'.


A typical git day for the maintainer implements the above policy
by doing the following:

 - Scan mailing list and #git channel log.  Respond with review
   comments, suggestions etc.  Kibitz.  Collect potentially
   usable patches from the mailing list.  Patches about a single
   topic go to one mailbox (I read my mail in Gnus, and type
   \C-o to save/append messages in files in mbox format).

 - Review the patches in the saved mailboxes.  Edit proposed log
   message for typofixes and clarifications, and add Acks
   collected from the list.  Edit patch to incorporate "Oops,
   that should have been like this" fixes from the discussion.

 - Classify the collected patches and handle 'master' and
   'maint' updates:

   - Obviously correct fixes that pertain to the tip of 'maint'
     are directly applied to 'maint'.

   - Obviously correct fixes that pertain to the tip of 'master'
     are directly applied to 'master'.

   This step is done with "git am".

     $ git checkout maint    ;# or "git checkout maint"
     $ git am -3 -s mailbox
     $ make test

 - Merge downwards (maint->master):

     $ git checkout master
     $ git merge maint
     $ make test

 - Review the last issue of "What's cooking" message, review the
   topics scheduled for merging upwards (topic->master and
   topic->maint), and merge.

     $ git checkout master    ;# or "git checkout maint"
     $ git merge ai/topic     ;# or "git merge ai/maint-topic"
     $ git log -p ORIG_HEAD.. ;# final review
     $ git diff ORIG_HEAD..   ;# final review
     $ make test              ;# final review
     $ git branch -d ai/topic ;# or "git branch -d ai/maint-topic"

 - Merge downwards (maint->master) if needed:

     $ git checkout master
     $ git merge maint
     $ make test

 - Merge downwards (master->next) if needed:

     $ git checkout next
     $ git merge master
     $ make test

 - Handle the remaining patches:

   - Anything unobvious that is applicable to 'master' (in other
     words, does not depend on anything that is still in 'next'
     and not in 'master') is applied to a new topic branch that
     is forked from the tip of 'master'.  This includes both
     enhancements and unobvious fixes to 'master'.  A topic
     branch is named as ai/topic where "ai" is typically
     author's initial and "topic" is a descriptive name of the
     topic (in other words, "what's the series is about").

   - An unobvious fix meant for 'maint' is applied to a new
     topic branch that is forked from the tip of 'maint'.  The
     topic is named as ai/maint-topic.

   - Changes that pertain to an existing topic are applied to
     the branch, but:

     - obviously correct ones are applied first;

     - questionable ones are discarded or applied to near the tip;

   - Replacement patches to an existing topic are accepted only
     for commits not in 'next'.

   The above except the "replacement" are all done with:

     $ git am -3 -s mailbox

   while patch replacement is often done by:

     $ git format-patch ai/topic~$n..ai/topic ;# export existing

   then replace some parts with the new patch, and reapplying:

     $ git reset --hard ai/topic~$n
     $ git am -3 -s 000*.txt

   The full test suite is always run for 'maint' and 'master'
   after patch application; for topic branches the tests are run
   as time permits.

 - Update "What's cooking" message to review the updates to
   existing topics, newly added topics and graduated topics.

   This step is helped with Meta/UWC script.

 - Merge topics to 'next'.  For each branch whose tip is not
   merged to 'next', one of three things can happen:

   - The commits are all next-worthy; merge the topic to next:

     $ git checkout next
     $ git merge ai/topic     ;# or "git merge ai/maint-topic"
     $ make test

   - The new parts are of mixed quality, but earlier ones are
     next-worthy; merge the early parts to next:
 
     $ git checkout next
     $ git merge ai/topic~2   ;# the tip two are dubious
     $ make test

   - Nothing is next-worthy; do not do anything.

 - Rebase topics that do not have any commit in next yet.  This
   step is optional but sometimes is worth doing when an old
   series that is not in next can take advantage of low-level
   framework change that is merged to 'master' already.

     $ git rebase master ai/topic

   This step is helped with Meta/git-topic.perl script to
   identify which topic is rebaseable.  There also is a
   pre-rebase hook to make sure that topics that are already in
   'next' are not rebased beyond the merged commit.

 - Rebuild "pu" to merge the tips of topics not in 'next'.

     $ git checkout pu
     $ git reset --hard next
     $ git merge ai/topic     ;# repeat for all remaining topics
     $ make test

   This step is helped with Meta/PU script

 - Push four integration branches to a private repository at
   k.org and run "make test" on all of them.

 - Push four integration branches to /pub/scm/git/git.git at
   k.org.  This triggers its post-update hook which:

    (1) runs "git pull" in $HOME/git-doc/ repository to pull
        'master' just pushed out;

    (2) runs "make doc" in $HOME/git-doc/, install the generated
        documentation in staging areas, which are separate
        repositories that have html and man branches checked
        out.

    (3) runs "git commit" in the staging areas, and run "git
        push" back to /pub/scm/git/git.git/ to update the html
        and man branches.

    (4) installs generated documentation to /pub/software/scm/git/docs/
        to be viewed from http://www.kernel.org/

 - Fetch html and man branches back from k.org, and push four
   integration branches and the two documentation branches to
   repo.or.cz


Some observations to be made.

 * Each topic is tested individually, and also together with
   other topics cooking in 'next'.  Until it matures, none part
   of it is merged to 'master'.

 * A topic already in 'next' can get fixes while still in
   'next'.  Such a topic will have many merges to 'next' (in
   other words, "git log --first-parent next" will show many
   "Merge ai/topic to next" for the same topic.

 * An unobvious fix for 'maint' is cooked in 'next' and then
   merged to 'master' to make extra sure it is Ok and then
   merged to 'maint'.

 * Even when 'next' becomes empty (in other words, all topics
   prove stable and are merged to 'master' and "git diff master
   next" shows empty), it has tons of merge commits that will
   never be in 'master'.

 * In principle, "git log --first-parent master..next" should
   show nothing but merges (in practice, there are fixup commits
   and reverts that are not merges).

 * Commits near the tip of a topic branch that are not in 'next'
   are fair game to be discarded, replaced or rewritten.
   Commits already merged to 'next' will not be.

 * Being in the 'next' branch is not a guarantee for a topic to
   be included in the next feature release.  Being in the
   'master' branch typically is.

Re: Addendum to "MaintNotes"

[Jeff King]

Thanks for writing this, Junio. It is always interesting to see git
workflows discussed in detail, and yours is more interesting than most.

On Wed, Nov 21, 2007 at 04:32:55PM -0800, Junio C Hamano wrote:

>    - Obviously correct fixes that pertain to the tip of 'master'
>      are directly applied to 'master'.
> 
>    This step is done with "git am".
> 
>      $ git checkout maint    ;# or "git checkout maint"

One of those "maints" should probably be "master".

-Peff

Re: Addendum to "MaintNotes"

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Thanks for writing this, Junio. It is always interesting to see git
> workflows discussed in detail, and yours is more interesting than most.

Now I've written this down, I can sleep well at night, knowing
that I can safely be hit by a bus any day ;-)

> On Wed, Nov 21, 2007 at 04:32:55PM -0800, Junio C Hamano wrote:
>
>>    - Obviously correct fixes that pertain to the tip of 'master'
>>      are directly applied to 'master'.
>> 
>>    This step is done with "git am".
>> 
>>      $ git checkout maint    ;# or "git checkout maint"
>
> One of those "maints" should probably be "master".

Heh, thanks.  I am hoping somebody do grammartical copyediting
and throw back a patch to add it in Documentation/howto.

Re: Addendum to "MaintNotes"

[David Kastrup]

Junio C Hamano <gitster@pobox.com> writes:

> Jeff King <peff@peff.net> writes:
>
>> Thanks for writing this, Junio. It is always interesting to see git
>> workflows discussed in detail, and yours is more interesting than most.
>
> Now I've written this down, I can sleep well at night, knowing
> that I can safely be hit by a bus any day ;-)

So regarding git development, a bus error will no longer imply a core
dump?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Addendum to "MaintNotes"

[Jeff King]

On Thu, Nov 22, 2007 at 02:42:20AM -0800, Junio C Hamano wrote:

> Heh, thanks.  I am hoping somebody do grammartical copyediting
> and throw back a patch to add it in Documentation/howto.

I just read it over and that was the only error I noticed (though there
might be conceptual errors or confusing things that others could point
out better -- I actually knew a lot of this information already. At
least I'm one extra pair of eyes).

Patch is below. Besides the intro, the diff from your original is:

@@ -90,7 +99,7 @@ by doing the following:
 
    This step is done with "git am".
 
-     $ git checkout maint    ;# or "git checkout maint"
+     $ git checkout master    ;# or "git checkout maint"
      $ git am -3 -s mailbox
      $ make test
 
@@ -168,7 +177,8 @@ by doing the following:
  - Update "What's cooking" message to review the updates to
    existing topics, newly added topics and graduated topics.
 
-   This step is helped with Meta/UWC script.
+   This step is helped with Meta/UWC script (where Meta/ contains
+   a checkout of the 'todo' branch).
 
  - Merge topics to 'next'.  For each branch whose tip is not
    merged to 'next', one of three things can happen:

-- >8 --
From: Junio C Hamano <gitster@pobox.com>

Add "how to maintain git" document. Foreward by Jeff King.

---
diff --git a/Documentation/howto/maintain-git.txt b/Documentation/howto/maintain-git.txt
new file mode 100644
index 0000000..159a0a3
--- /dev/null
+++ b/Documentation/howto/maintain-git.txt
@@ -0,0 +1,278 @@
+How to maintain git
+===================
+
+Imagine that git development is racing along as usual, when our friendly
+neighborhood maintainer is struck down by a wayward bus. Out of the
+hordes of suckers^W loyal developers, you have been tricked^W chosen to
+step up as the new maintainer. This howto will show you "how to" do it.
+
+-- >8 --
+The maintainer's git time is spent on three activities.
+
+ - Communication (60%)
+
+   Mailing list discussions on general design, fielding user
+   questions, diagnosing bug reports; reviewing, commenting on,
+   suggesting alternatives to, and rejecting patches.
+
+ - Integration (30%)
+
+   Applying new patches from the contributors while spotting and
+   correcting minor mistakes, shuffling the integration and
+   testing branches, pushing the results out, cutting the
+   release, and making announcements.
+
+ - Own development (10%)
+
+   Scratching my own itch and sending proposed patch series out.
+
+The policy on Integration is informally mentioned in "A Note
+from the maintainer" message, which is periodically posted to
+this mailing list after each feature release is made.
+
+The policy.
+
+ - Feature releases are numbered as vX.Y.Z and are meant to
+   contain bugfixes and enhancements in any area, including
+   functionality, performance and usability, without regression.
+
+ - Maintenance releases are numbered as vX.Y.Z.W and are meant
+   to contain only bugfixes for the corresponding vX.Y.Z feature
+   release and earlier maintenance releases vX.Y.Z.V (V < W).
+
+ - 'master' branch is used to prepare for the next feature
+   release. In other words, at some point, the tip of 'master'
+   branch is tagged with vX.Y.Z.
+
+ - 'maint' branch is used to prepare for the next maintenance
+   release.  After the feature release vX.Y.Z is made, the tip
+   of 'maint' branch is set to that release, and bugfixes will
+   accumulate on the branch, and at some point, the tip of the
+   branch is tagged with vX.Y.Z.1, vX.Y.Z.2, and so on.
+
+ - 'next' branch is used to publish changes (both enhancements
+   and fixes) that (1) have worthwhile goal, (2) are in a fairly
+   good shape suitable for everyday use, (3) but have not yet
+   demonstrated to be regression free.  New changes are tested
+   in 'next' before merged to 'master'.
+
+ - 'pu' branch is used to publish other proposed changes that do
+   not yet pass the criteria set for 'next'.
+
+ - The tips of 'master', 'maint' and 'next' branches will always
+   fast forward, to allow people to build their own
+   customization on top of them.
+
+ - Usually 'master' contains all of 'maint', 'next' contains all
+   of 'master' and 'pu' contains all of 'next'.
+
+ - The tip of 'master' is meant to be more stable than any
+   tagged releases, and the users are encouraged to follow it.
+
+ - The 'next' branch is where new action takes place, and the
+   users are encouraged to test it so that regressions and bugs
+   are found before new topics are merged to 'master'.
+
+
+A typical git day for the maintainer implements the above policy
+by doing the following:
+
+ - Scan mailing list and #git channel log.  Respond with review
+   comments, suggestions etc.  Kibitz.  Collect potentially
+   usable patches from the mailing list.  Patches about a single
+   topic go to one mailbox (I read my mail in Gnus, and type
+   \C-o to save/append messages in files in mbox format).
+
+ - Review the patches in the saved mailboxes.  Edit proposed log
+   message for typofixes and clarifications, and add Acks
+   collected from the list.  Edit patch to incorporate "Oops,
+   that should have been like this" fixes from the discussion.
+
+ - Classify the collected patches and handle 'master' and
+   'maint' updates:
+
+   - Obviously correct fixes that pertain to the tip of 'maint'
+     are directly applied to 'maint'.
+
+   - Obviously correct fixes that pertain to the tip of 'master'
+     are directly applied to 'master'.
+
+   This step is done with "git am".
+
+     $ git checkout master    ;# or "git checkout maint"
+     $ git am -3 -s mailbox
+     $ make test
+
+ - Merge downwards (maint->master):
+
+     $ git checkout master
+     $ git merge maint
+     $ make test
+
+ - Review the last issue of "What's cooking" message, review the
+   topics scheduled for merging upwards (topic->master and
+   topic->maint), and merge.
+
+     $ git checkout master    ;# or "git checkout maint"
+     $ git merge ai/topic     ;# or "git merge ai/maint-topic"
+     $ git log -p ORIG_HEAD.. ;# final review
+     $ git diff ORIG_HEAD..   ;# final review
+     $ make test              ;# final review
+     $ git branch -d ai/topic ;# or "git branch -d ai/maint-topic"
+
+ - Merge downwards (maint->master) if needed:
+
+     $ git checkout master
+     $ git merge maint
+     $ make test
+
+ - Merge downwards (master->next) if needed:
+
+     $ git checkout next
+     $ git merge master
+     $ make test
+
+ - Handle the remaining patches:
+
+   - Anything unobvious that is applicable to 'master' (in other
+     words, does not depend on anything that is still in 'next'
+     and not in 'master') is applied to a new topic branch that
+     is forked from the tip of 'master'.  This includes both
+     enhancements and unobvious fixes to 'master'.  A topic
+     branch is named as ai/topic where "ai" is typically
+     author's initial and "topic" is a descriptive name of the
+     topic (in other words, "what's the series is about").
+
+   - An unobvious fix meant for 'maint' is applied to a new
+     topic branch that is forked from the tip of 'maint'.  The
+     topic is named as ai/maint-topic.
+
+   - Changes that pertain to an existing topic are applied to
+     the branch, but:
+
+     - obviously correct ones are applied first;
+
+     - questionable ones are discarded or applied to near the tip;
+
+   - Replacement patches to an existing topic are accepted only
+     for commits not in 'next'.
+
+   The above except the "replacement" are all done with:
+
+     $ git am -3 -s mailbox
+
+   while patch replacement is often done by:
+
+     $ git format-patch ai/topic~$n..ai/topic ;# export existing
+
+   then replace some parts with the new patch, and reapplying:
+
+     $ git reset --hard ai/topic~$n
+     $ git am -3 -s 000*.txt
+
+   The full test suite is always run for 'maint' and 'master'
+   after patch application; for topic branches the tests are run
+   as time permits.
+
+ - Update "What's cooking" message to review the updates to
+   existing topics, newly added topics and graduated topics.
+
+   This step is helped with Meta/UWC script (where Meta/ contains
+   a checkout of the 'todo' branch).
+
+ - Merge topics to 'next'.  For each branch whose tip is not
+   merged to 'next', one of three things can happen:
+
+   - The commits are all next-worthy; merge the topic to next:
+
+     $ git checkout next
+     $ git merge ai/topic     ;# or "git merge ai/maint-topic"
+     $ make test
+
+   - The new parts are of mixed quality, but earlier ones are
+     next-worthy; merge the early parts to next:
+
+     $ git checkout next
+     $ git merge ai/topic~2   ;# the tip two are dubious
+     $ make test
+
+   - Nothing is next-worthy; do not do anything.
+
+ - Rebase topics that do not have any commit in next yet.  This
+   step is optional but sometimes is worth doing when an old
+   series that is not in next can take advantage of low-level
+   framework change that is merged to 'master' already.
+
+     $ git rebase master ai/topic
+
+   This step is helped with Meta/git-topic.perl script to
+   identify which topic is rebaseable.  There also is a
+   pre-rebase hook to make sure that topics that are already in
+   'next' are not rebased beyond the merged commit.
+
+ - Rebuild "pu" to merge the tips of topics not in 'next'.
+
+     $ git checkout pu
+     $ git reset --hard next
+     $ git merge ai/topic     ;# repeat for all remaining topics
+     $ make test
+
+   This step is helped with Meta/PU script
+
+ - Push four integration branches to a private repository at
+   k.org and run "make test" on all of them.
+
+ - Push four integration branches to /pub/scm/git/git.git at
+   k.org.  This triggers its post-update hook which:
+
+    (1) runs "git pull" in $HOME/git-doc/ repository to pull
+        'master' just pushed out;
+
+    (2) runs "make doc" in $HOME/git-doc/, install the generated
+        documentation in staging areas, which are separate
+        repositories that have html and man branches checked
+        out.
+
+    (3) runs "git commit" in the staging areas, and run "git
+        push" back to /pub/scm/git/git.git/ to update the html
+        and man branches.
+
+    (4) installs generated documentation to /pub/software/scm/git/docs/
+        to be viewed from http://www.kernel.org/
+
+ - Fetch html and man branches back from k.org, and push four
+   integration branches and the two documentation branches to
+   repo.or.cz
+
+
+Some observations to be made.
+
+ * Each topic is tested individually, and also together with
+   other topics cooking in 'next'.  Until it matures, none part
+   of it is merged to 'master'.
+
+ * A topic already in 'next' can get fixes while still in
+   'next'.  Such a topic will have many merges to 'next' (in
+   other words, "git log --first-parent next" will show many
+   "Merge ai/topic to next" for the same topic.
+
+ * An unobvious fix for 'maint' is cooked in 'next' and then
+   merged to 'master' to make extra sure it is Ok and then
+   merged to 'maint'.
+
+ * Even when 'next' becomes empty (in other words, all topics
+   prove stable and are merged to 'master' and "git diff master
+   next" shows empty), it has tons of merge commits that will
+   never be in 'master'.
+
+ * In principle, "git log --first-parent master..next" should
+   show nothing but merges (in practice, there are fixup commits
+   and reverts that are not merges).
+
+ * Commits near the tip of a topic branch that are not in 'next'
+   are fair game to be discarded, replaced or rewritten.
+   Commits already merged to 'next' will not be.
+
+ * Being in the 'next' branch is not a guarantee for a topic to
+   be included in the next feature release.  Being in the
+   'master' branch typically is.

Re: Addendum to "MaintNotes"

[Jeff King]

On Thu, Nov 22, 2007 at 11:48:09AM +0100, David Kastrup wrote:

> > Now I've written this down, I can sleep well at night, knowing
> > that I can safely be hit by a bus any day ;-)
> 
> So regarding git development, a bus error will no longer imply a core
> dump?

I'm slightly ashamed to admit to laughing at that.

-Peff