Recommended work flow with git to send in patches

Recommended work flow with git to send in patches

[Tong Sun]

Hi,

Compressing my "life long story" into a single question -- what's the
recommended work flow to work with git and send in patches, when
upstream might be slow in respond, and require squashing relevant
patches into one?

You can use my following message as a start point, and please answer
my last question, which I've been asking twice (in different ways)
with no answer.

Please CC me when replying.

Thanks

---------- Forwarded message ----------
From: Tong Sun <suntong@cpan.org>
Date: Sun, Jun 6, 2010 at 8:56 PM
Subject: Working with git and sending in patches
To: grml-devel@ml.grml.org


Hi,

Just trying to put all jigsaw puzzle together here. Please correct me
if I'm wrong.

First of all, philosophy for version control with git:

. While developing, small/independent commits are good thing, so that
it's easy to decouple different changes.

. But when integrating something in a main branch, commits should contain all
logical/related changes.

Steps (using grml-debootstrap as an example):

- do initial git pull into grml-debootstrap

  git pull git://git.grml.org/grml-debootstrap master

- Go into grml-debootstrap and start a new branch

  git checkout -b t/my-working-branch

- work on the code, commit, hack, commit, hack, commit -- commit often
& commit small

- when AOK and need to integrate patches into main branch, squash all
patches into one

  git rebase -i origin/master

- send in patches via email (to grml-devel@ml.grml.org)

  git format-patch origin
  git send-email --to grml-devel@ml.grml.org ...

Please correct me if anything above is wrong.

Now, question, having done above, if I start to work some logically
unrelated patches, what steps should I take? (I don't want 'git
rebase' to pick up patches that I've already sent in).

Re: Recommended work flow with git to send in patches

[Ævar Arnfjörð Bjarmason]

On Tue, Jul 27, 2010 at 15:31, Tong Sun <suntong@cpan.org> wrote:
> - work on the code, commit, hack, commit, hack, commit -- commit often
> & commit small

So far so good.

> - when AOK and need to integrate patches into main branch, squash all
> patches into one

That's just silly. Why is the policy to destroy commit information
when submitting patches?

Re: Recommended work flow with git to send in patches

[Ramkumar Ramachandra]

Hi Tong,

Tong Sun writes:
> Compressing my "life long story" into a single question -- what's the
> recommended work flow to work with git and send in patches, when
> upstream might be slow in respond, and require squashing relevant
> patches into one?

Personally, I use `git symbolic-ref` to create a new branch without
history, stage and create commits to send off to the list from there;
it's also worth noting that I keep the branch to fixup my commits and
re-roll after reviews.

-- Ram

Re: Recommended work flow with git to send in patches

[Tong Sun]

On Tue, Jul 27, 2010 at 11:35 AM, Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> On Tue, Jul 27, 2010 at 15:31, Tong Sun <suntong@cpan.org> wrote:
>> - work on the code, commit, hack, commit, hack, commit -- commit often
>> & commit small
>
> So far so good.
>
>> - when AOK and need to integrate patches into main branch, squash all
>> patches into one
>
> That's just silly. Why is the policy to destroy commit information
> when submitting patches?

Sorry about the confusing. The original message was composed under the
condition that all patches are relevant patches that work together to
form a big logical changes. I was required to do the squashing.

thanks

Re: Recommended work flow with git to send in patches

[Tong Sun]

On Tue, Jul 27, 2010 at 11:39 AM, Ramkumar Ramachandra
<artagnon@gmail.com> wrote:

>> Compressing my "life long story" into a single question -- what's the
>> recommended work flow to work with git and send in patches, when
>> upstream might be slow in respond, and require squashing relevant
>> patches into one?
>
> Personally, I use `git symbolic-ref` to create a new branch without
> history, stage and create commits to send off to the list from there;
> it's also worth noting that I keep the branch to fixup my commits and
> re-roll after reviews.

Thanks a lot for the comment.

Could you elaborate it a bit with actual commands, starting from 'git
pull git://remote/project master' please?

I'm new to git and the above comment barely helps me to put all jigsaw
puzzle together.

thanks

Re: Recommended work flow with git to send in patches

[Ramkumar Ramachandra]

Hi Tong,

Tong Sun writes:
> Thanks a lot for the comment.
> 
> Could you elaborate it a bit with actual commands, starting from 'git
> pull git://remote/project master' please?

$ git symbolic-ref HEAD refs/heads/rollout
$ rm .git/index # Git recreates it automatically anyway
$ git add LICENSE
$ git commit -m "Add LICENSE" # root commit
$ # git add ... git commit cycle

-- Ram

Re: Recommended work flow with git to send in patches

[Jakub Narebski]

Ramkumar Ramachandra <artagnon@gmail.com> writes:
> Tong Sun writes:
> > Thanks a lot for the comment.
> > 
> > Could you elaborate it a bit with actual commands, starting from 'git
> > pull git://remote/project master' please?
> 
> $ git symbolic-ref HEAD refs/heads/rollout
> $ rm .git/index # Git recreates it automatically anyway
> $ git add LICENSE
> $ git commit -m "Add LICENSE" # root commit
> $ # git add ... git commit cycle

Ramkumar, in git 1.7.2 you can simply use "git checkout --orphan newbranch".

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: Recommended work flow with git to send in patches

[Ramkumar Ramachandra]

Hi Jakub,

Jakub Narebski writes:
> Ramkumar Ramachandra <artagnon@gmail.com> writes:
> > $ git symbolic-ref HEAD refs/heads/rollout
> > $ rm .git/index # Git recreates it automatically anyway
> > $ git add LICENSE
> > $ git commit -m "Add LICENSE" # root commit
> > $ # git add ... git commit cycle
> 
> Ramkumar, in git 1.7.2 you can simply use "git checkout --orphan newbranch".

Thanks for pointing this out! I'm still using many plubming tools- I
ought to find out their equivalents in the new porcelain.

-- Ram

Re: Recommended work flow with git to send in patches

[Jakub Narebski]

Tong Sun <suntong@cpan.org> writes:

> Just trying to put all jigsaw puzzle together here. Please correct me
> if I'm wrong.
> 
> First of all, philosophy for version control with git:
> 
> . While developing, small/independent commits are good thing, so that
>   it's easy to decouple different changes.
> 
> . But when integrating something in a main branch, commits should contain all
>   logical/related changes.

I think that in final results, i.e. in patches that you send, or
commits that you send pull request for, you should have commits that
do one thing, and do it completely and without breaking.  Nevertheless
having small commits that you publish / send to maintainer is a good
thing; it is always easy to review a few small patches, than one
mage-patch.

> 
> Steps (using grml-debootstrap as an example):
> 
> - do initial git pull into grml-debootstrap
> 
>   git pull git://git.grml.org/grml-debootstrap master

Why not git-clone (possibly shallow, if you are working on one-shot
patch or patch series)?

If you plan to continue working on this repository, and it is not
one-shot patch or patch series, it would be better (easier in the
future) to use "git remote add".

> 
> - Go into grml-debootstrap and start a new branch
> 
>   git checkout -b t/my-working-branch
> 
> - work on the code, commit, hack, commit, hack, commit -- commit often
>   & commit small

You can always use 'git commit' + 'git commit --amend' if you want to
fix previous commit, instead of creating new commit.

> 
> - when AOK and need to integrate patches into main branch, squash all
>   patches into one
> 
>   git rebase -i origin/master

Reorder, edit, squash patches.

> 
> - send in patches via email (to grml-devel@ml.grml.org)
> 
>   git format-patch origin

With larger patch series, it could possibly be of the form:

    mkdir patches/
    git format-patch -o patches/ --cover-letter
    [edit cover letter, filling in template]
    [edit patches if necessary, adding comments between "---" and diffstat]

>   git send-email --to grml-devel@ml.grml.org ...
> 
> Please correct me if anything above is wrong.

You can also use some patch management interface, like StGit, Guilt or
TopGit.  I personally use StGit, so the above description is modified
in that there is 'stg init', there is 'stg new some-patch' and possibly
multiple 'stg refresh' when working on commit, there might be 'stg goto's
and 'stg push'es and 'stg pop's to go back and forth between patches,
and fix them.

It's a matter of taste whether to use some kind of patch management /
/ mail queue interface, or to use interactive rebase instead.

> Now, question, having done above, if I start to work some logically
> unrelated patches, what steps should I take? (I don't want 'git
> rebase' to pick up patches that I've already sent in).

If you want to start to work on some logically unrelated patches, you
should start a new branch for that.

Alternatively, when some or all patches are accepted upstream, you
download new changes using 'git pull' or 'git fetch' or 'git remote
update', and then rebase your branch on top of branch it is based on,
or to be more exact on top of branch your patches are now in.  'git
rebase' (or 'git pull --rebase' or even 'git pull', if configured
appropriately) would automatically skip patches that got applied
(sometimes you need to tell git to 'git rebase --skip' if it didn't
detect this).

It would be 'stg rebase <base branch>' (usually <base branch> is
'origin'='origin/master' or 'master'), and 'stg clean -a' to remove
empty applied patches.


P.S. Doesn't GRML have web page / wiki page for developers?
     http://grml.org/git/

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: Recommended work flow with git to send in patches

[Tong Sun]

On Tue, Jul 27, 2010 at 1:35 PM, Jakub Narebski <jnareb@gmail.com> wrote:

> Doesn't GRML have web page / wiki page for developers?

Yes, http://grml.org/git/, but it is kind of reference book style,
doesn't cover much on the work flow, especially on the topic of
no-writing-privilege user contributing back.

>> - do initial git pull into grml-debootstrap
>>
>>   git pull git://git.grml.org/grml-debootstrap master
>
> Why not git-clone (possibly shallow, if you are working on one-shot
> patch or patch series)?

Ok, to explain it, I have to touch upon my "life long story" of using
git. Long story short on this, the recommended work flow that I
searched and found from the Inet was to do 'git clone' from web then
'git clone' a local working copy. Here is my trying log:

# Download the latest version of the repository without downloading all the
history, using "shallow checkouts".

  git clone --depth 1 git://git.grml.org/grml-debootstrap.git

create working repo:

  $ git-clone --depth 1 grml-debootstrap grml-debootstrap.working
  Initialized empty Git repository in
/export/repositories/gitwork/grml/grml-debootstrap.working/.git/
  fatal: attempt to fetch/clone from a shallow repository
  ^^^^^^^^^^^^^^^

Seeing that fatal error, and not knowing where to get help from, I
just gave up the 'git clone' approach. Please be specific (with git
commands), how would I use 'git clone' for working on one-shot patch
or patch series.

Got to run, will comment on the rest later.

thanks

Re: Recommended work flow with git to send in patches

[Jakub Narebski]

On Tue, Jul 27, 2010, Tong Sun wrote:
> On Tue, Jul 27, 2010 at 1:35 PM, Jakub Narebski <jnareb@gmail.com> wrote:
> 
> > Doesn't GRML have web page / wiki page for developers?
> 
> Yes, http://grml.org/git/, but it is kind of reference book style,
> doesn't cover much on the work flow, especially on the topic of
> no-writing-privilege user contributing back.

You could point lack of information for "leaf" contributors to them...

> > > - do initial git pull into grml-debootstrap
> > >
> > >   git pull git://git.grml.org/grml-debootstrap master
> >
> > Why not git-clone (possibly shallow, if you are working on one-shot
> > patch or patch series)?
> 
> Ok, to explain it, I have to touch upon my "life long story" of using
> git. Long story short on this, the recommended work flow that I
> searched and found from the Inet was to do 'git clone' from web then
> 'git clone' a local working copy. Here is my trying log:
> 
> # Download the latest version of the repository without downloading all the
> history, using "shallow checkouts".
> 
>   git clone --depth 1 git://git.grml.org/grml-debootstrap.git

O.K.
 
> create working repo:
> 
>   $ git-clone --depth 1 grml-debootstrap grml-debootstrap.working
>   Initialized empty Git repository in
> /export/repositories/gitwork/grml/grml-debootstrap.working/.git/
>   fatal: attempt to fetch/clone from a shallow repository
>   ^^^^^^^^^^^^^^^
> 
> Seeing that fatal error, and not knowing where to get help from, I
> just gave up the 'git clone' approach. Please be specific (with git
> commands), how would I use 'git clone' for working on one-shot patch
> or patch series.

I don't understand this second step.  Why do you want this second clone?
It is totally unnecessary.  The clone you make is working repository
you can make your contributions in.

In short, the workflow should look like this (I use here grml as example,
but in practice I used git repository as I know it is fairly small, so
the numbers, SHA-1 identifiers and branches won't match)

  $ cd <somewhere>
  $ git clone --depth 1 git://git.grml.org/grml-debootstrap.git
  Initialized empty Git repository in <somwehere>/grml-debootstrap/.git/
  [...]
  $ cd grml-debootstrap
  $ git branch -a
  * master
    remotes/origin/HEAD -> origin/master
    remotes/origin/master
  [...]

Now you have two choices: you can work on 'master' branch (reasonable
if you want to do only single thing), or you can create your own branch
(one branch for one independent unrelated feature).  Let's create new
branch off 'origin' == 'origin/master' == 'remotes/origin/master'

  $ git checkout -b t/my-working-branch origin
  Branch t/my-working-branch set up to track remote branch master from origin.
  Switched to a new branch 't/my-working-branch'

We could first setup git in such way, that it sets up to track upstream
via rebasing rather than via merge (using `branch.autosetuprebase` config
variable), but let's not complicate matters.

Now you work on branch

  [edit, edit, edit]
  [commit or commit --amend]
  [repeat as necessary]
  
Now, after some time you feel that your changes are ready for sending.
First, download any new changes, if any.

  $ git fetch

Then use interactive rebase to both update your changes to apply to most
recent upstream code, and also reorder and edit your patches (e.g. fix
typo in commit message, move fix commit and squash it with fixed commit,
etc.)

  $ git rebase --interactive origin
  [...]
  Successfully rebased and updated refs/heads/t/my-working-branch.

Then you need to generate patches.  If its more of them, I found it good
idea to save them in separate subdirectory, but it might be not necessary
for you

  $ git format-patch --cover-letter origin..

There are some situations where you don't need cover letter, e.g. if you
are sending single patch, but they are quite useful especially with longer
series.

  [edit '0000-cover-letter.patch' at least]
  [add comments to individual patches, if necessary]

Then you can either use your email client, or just

  $ git send-email --dry-run *.patch

(Here you see why saving patch series to separate subdirectory by
using '-o' option makes it easy to pick up relevant patches... ;-))

HTH
-- 
Jakub Narebski
Poland

Re: Recommended work flow with git to send in patches

[Tong Sun]

On Tue, Jul 27, 2010 at 1:35 PM, Jakub Narebski <jnareb@gmail.com> wrote:

>> First of all, philosophy for version control with git:
>>
>> . While developing, small/independent commits are good thing, so that
>>   it's easy to decouple different changes.
>>
>> . But when integrating something in a main branch, commits should contain all
>>   logical/related changes.
>
> I think that in final results, i.e. in patches that you send, or
> commits that you send pull request for, you should have commits that
> do one thing, and do it completely and without breaking.  Nevertheless
> having small commits that you publish / send to maintainer is a good
> thing; it is always easy to review a few small patches, than one
> mage-patch.

Yeah, that's actually exactly what I believed before getting feedbacks
from grml developers for squashed patches. It's an interesting topic
to me, so let's dig deeper into it.

Say that I need to add a feature to a CLI program. I would
instinctively divide it into 3 logical steps/patches, 1st to the user
interface (the command line handling), 2nd to the implementation, and
3rd to the document.

Do you think 3 small patches is the way to go, or a single patch is,
since all 3 are logically related?

Now back to our topic, thanks for your work flow explanation, I'll
answer/ask in this single message.

> Why not git-clone (possibly shallow, if you are working on one-shot
> patch or patch series)?
>> Ok, to explain it, I have to touch upon my "life long story" of using
>> git.

> I don't understand this second step.  Why do you want this second clone?

That's what I searched and found from the Inet when I was looking for
the recommended work flow, which was to do 'git clone' from web once
then 'git clone' several local working copies to work on several
independent unrelated features. Now I know creating my own local
branches is the way to go.

> If you plan to continue working on this repository, and it is not
> one-shot patch or patch series, it would be better (easier in the
> future) to use "git remote add".

Could you elaborate more on this with git commands please, so that I
can have a full picture?

Thanks again for your clearly explanations, I think I don't any
further questions for the moment.

cheers,

Re: Recommended work flow with git to send in patches

[Jakub Narebski]

On Thu, Jul 29, 2010, Tong Sun wrote:
> On Tue, Jul 27, 2010 at 1:35 PM, Jakub Narebski <jnareb@gmail.com> wrote:
> 
>>> First of all, philosophy for version control with git:
>>>
>>> . While developing, small/independent commits are good thing, so that
>>>   it's easy to decouple different changes.
>>>
>>> . But when integrating something in a main branch, commits should contain all
>>>   logical/related changes.
>>
>> I think that in final results, i.e. in patches that you send, or
>> commits that you send pull request for, you should have commits that
>> do one thing, and do it completely and without breaking.  Nevertheless
>> having small commits that you publish / send to maintainer is a good
>> thing; it is always easy to review a few small patches, than one
>> mage-patch.
> 
> Yeah, that's actually exactly what I believed before getting feedbacks
> from grml developers for squashed patches. It's an interesting topic
> to me, so let's dig deeper into it.
> 
> Say that I need to add a feature to a CLI program. I would
> instinctively divide it into 3 logical steps/patches, 1st to the user
> interface (the command line handling), 2nd to the implementation, and
> 3rd to the document.
> 
> Do you think 3 small patches is the way to go, or a single patch is,
> since all 3 are logically related?

In this situation it is obvious to me that you should send single
squashed patch, as otherwise commit does not contain all related and
connected changes: command line handling without implementation doesn't
make sense, and if you are implementing something, you should document
it.


An example of change that might, or might be not split into two commits
is simple bug fix.  You can either first write failing test showing the
breakage, and then write fix and change test expectation to pass, or you
can write fix and test in one commit.

Here are a few generalized examples from git repository history where
you might want to send a change as a series of patches rather than in
one large single patch:

1. If you are improving documentation, you can split your patch into
   one adding missing documentation for some feature, and second adding
   examples of using said feature.

2. If you plan to make some part of code used more widely, you can in
   first commit make API public, in second add support for feature to
   wider codebase, and in third add tests for this support.

   Similar thing with having doing refactoring first, then using this
   refactoring to easy add new feature.

3. If you are fixing some compiler warnings, fixing each class/type
   of warnings could be made into separate commits.

4. You can have one commit adding feature, and second adding support
   for said feature (for parameters / subcommands that use said feature)
   to shell completion.

Etc.

> Now back to our topic, thanks for your work flow explanation, I'll
> answer/ask in this single message.
> 
>> Why not git-clone (possibly shallow, if you are working on one-shot
>> patch or patch series)?
>>>
>>> Ok, to explain it, I have to touch upon my "life long story" of using
>>> git.
> 
>> I don't understand this second step.  Why do you want this second clone?
> 
> That's what I searched and found from the Inet when I was looking for
> the recommended work flow, which was to do 'git clone' from web once
> then 'git clone' several local working copies to work on several
> independent unrelated features. Now I know creating my own local
> branches is the way to go.

I think "fork (clone) to branch" was from some very ancient git tutorials
(git has in-place branching and branch switching from time immemorial),
or from some (outdated?) Mercurial documentation.

That said shallow clone should be improved, so you can clone from
shallow clone with the same depth or shallower.  Current implementation
is a bit lacking.

>> If you plan to continue working on this repository, and it is not
>> one-shot patch or patch series, it would be better (easier in the
>> future) to use "git remote add".
> 
> Could you elaborate more on this with git commands please, so that I
> can have a full picture?
> 
> Thanks again for your clearly explanations, I think I don't any
> further questions for the moment.

I don't think that would apply in your situation, but "git remote add"
is used if you want to fetch changes from more than one upstream repository
(or you want to configure repository to push into).  This is an alternative
to one-shot "git pull <URL> <branch>" which does not save _any_ information
about upstream you fetched (pulled) from.

See git-remote manpage for details.
-- 
Jakub Narebski
Poland

Re: Recommended work flow with git to send in patches

[Tong Sun]

On Wed, Jul 28, 2010 at 7:20 PM, Jakub Narebski <jnareb@gmail.com> wrote:
>>> If you plan to continue working on this repository, and it is not
>>> one-shot patch or patch series, it would be better (easier in the
>>> future) to use "git remote add".
>>
> . . .
> I don't think that would apply in your situation, but "git remote add"
> is used if you want to fetch changes from more than one upstream repository
> (or you want to configure repository to push into).  This is an alternative
> to one-shot "git pull <URL> <branch>" which does not save _any_ information
> about upstream you fetched (pulled) from.

Got it.

> See git-remote manpage for details.

and http://www.kernel.org/pub/software/scm/git/docs/gittutorial.html as well.

thanks again. no more questions for now.

cheers