Git workflow: how to achieve?

Git workflow: how to achieve?

[George Spelvin]

Here's something I and my co-workers would like to achieve, but are not
too sure how to arrange.

I want to be committing to a feature branch, but always be compiling
and testing a merge of that branch and several others.  (Kind of like
linux-next.)  I want to be able to switch feature branches easily.

For example, I may have a background project that I'm working on slowly in
between short-term fixes.  Or I want to be running the latest kernel.org
kernel while my patches await approval.

If it's just my own projects, I can just commit in random order and
straighten things out later.  Although even that is problematic,
as I may not remember what line of development a cleanup patch is a
prerequisite for.  (This is something that darcs is apparently good at.)

But when I want to be testing something highly volatile like linux-next,
and ensuring that my work continues to merge with it cleanly, as well
as helping others with their branches, it becomes a daily pain.

The best attempt I have so far is to rebase a lot.  But that means that
I can't do any merging in my development branch lest the rebasing turn
into a mess.  And forcing everything to be linear makes changing branches
a pain.  And I can't share half-finished versions with co-workers.

This is all vaguely quilt-like, although I'd rather not worry about the
order of patches.  I suppose I'd like git to let me "commit under" the
final merge.  When I switch branches, git should reorganize the tree of
merges so that the current branch is only one merge from the HEAD.

(Another thing I've wished for that might be related is for a branch
to have a notion of its origin.  So I can rebase it onto an arbitrary
place in the commit tree without having to explicitly specify an origin.)

((Another really simple feature I keep wanting is "git commit -b
<newbranch>".  I should probably try to write a patch...))


Anyway, my feature ideas might be unworkable, and in any case, they'll
take a long time to implement.  Is there some easier way to achieve more
or less this effect?

Maybe the planned git-rebase improvements to handle merges better will
fix this, so I can just commit on top and periodically rebase the changes
under the head manually without too much pain?  (git rebase -i -p does
appear to be working better than I remember.)

H'm... in fact, it might be as easy as replacing "git pull" with
git rebase -p -i <last merge>^
(Delete the merge in the editor)
git pull <remote>

Annoying to remember, but not TOO bad.

Re: Git workflow: how to achieve?

[Uwe Kleine-König]

On Fri, Apr 24, 2009 at 12:58:43AM -0400, George Spelvin wrote:
> Here's something I and my co-workers would like to achieve, but are not
> too sure how to arrange.
> 
> I want to be committing to a feature branch, but always be compiling
> and testing a merge of that branch and several others.  (Kind of like
> linux-next.)  I want to be able to switch feature branches easily.
> 
> For example, I may have a background project that I'm working on slowly in
> between short-term fixes.  Or I want to be running the latest kernel.org
> kernel while my patches await approval.
> 
> If it's just my own projects, I can just commit in random order and
> straighten things out later.  Although even that is problematic,
> as I may not remember what line of development a cleanup patch is a
> prerequisite for.  (This is something that darcs is apparently good at.)
> 
> But when I want to be testing something highly volatile like linux-next,
> and ensuring that my work continues to merge with it cleanly, as well
> as helping others with their branches, it becomes a daily pain.
> 
> The best attempt I have so far is to rebase a lot.  But that means that
> I can't do any merging in my development branch lest the rebasing turn
> into a mess.  And forcing everything to be linear makes changing branches
> a pain.  And I can't share half-finished versions with co-workers.
If you develop a topic, say gs/fix_bug_12345, optimally you should base
it on the commit introducing the bug.  Then you can have several
integrating branches, say

	next/gs/fix_bug_12345
	linus/gs/fix_bug_12345
	$company_stable/gs/fix_bug_12345

and in these branches you can regularly merge in the current state of
your gs/fix_bug_12345 branch.  You might want to take a look on topgit,
which can automate the propagation for you.

And/or you might want to look into the linux-tip scripts by Ingo
Molnar[1].
 
> ((Another really simple feature I keep wanting is "git commit -b
> <newbranch>".  I should probably try to write a patch...))
Well currently this can be done in two commands:

	git checkout -b <newbranch>; git commit

AFAIK it even works if you already modified the index.

Best regards
Uwe

[1] git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git tip:.tip

-- 
Pengutronix e.K.                              | Uwe Kleine-König            |
Industrial Linux Solutions                    | http://www.pengutronix.de/  |

Re: Git workflow: how to achieve?

[Andreas Ericsson]

George Spelvin wrote:
> Here's something I and my co-workers would like to achieve, but are not
> too sure how to arrange.
> 
> I want to be committing to a feature branch, but always be compiling
> and testing a merge of that branch and several others.

At $dayjob we do a "build-on-push" thing, so every time a developer
feels his work is worthy of trying out for others, he ships it to the
mother-ship repository, where the update hook (should be post-receive,
but we haven't bothered changing it since it Just Works as is) kicks
the buildbot, triggering builds on all architectures and platforms
we support.

Assuming a static set of branches to merge, or a prefix or suffix
that marks the branches to merge, it wouldn't be difficult to let it
trigger "merge this and that" stuff as well.


>  I want to be able to switch feature branches easily.
> 

Well, git makes that absolutely trivial. No other vcs has as cheap
branching as git does.

> For example, I may have a background project that I'm working on slowly in
> between short-term fixes.  Or I want to be running the latest kernel.org
> kernel while my patches await approval.
> 
> If it's just my own projects, I can just commit in random order and
> straighten things out later.  Although even that is problematic,
> as I may not remember what line of development a cleanup patch is a
> prerequisite for.  (This is something that darcs is apparently good at.)
> 

Sounds like you'd benefit somewhat from using TopGit or some other
patch-queue management tool. I haven't used them myself, but I believe
this is the kind of problem they set out to solve.

> But when I want to be testing something highly volatile like linux-next,
> and ensuring that my work continues to merge with it cleanly, as well
> as helping others with their branches, it becomes a daily pain.
> 

Why? Just merge it to make sure it merges, and then throw away the result.
There's a post made by Linus somewhere on how he would like people to do
merges (ie, which to keep and which to throw away).

> The best attempt I have so far is to rebase a lot.  But that means that
> I can't do any merging in my development branch lest the rebasing turn
> into a mess.

Merging other branches into your development branch and keeping the result
is often a bad idea, since you can't later merge the development branch
without also merging everything that *you* merged. This makes it nearly
impossible to pick your feature into random spots in the DAG. If you ever
work anywhere but on the bleeding edge, you'll find merging the "main dev
branch" is a *really* bad idea.

>  And forcing everything to be linear makes changing branches
> a pain.

Everything needn't be linear, but if you can manage to keep each topic
linear that makes your series a lot easier to review.

>  And I can't share half-finished versions with co-workers.
> 

So finish it before you push. Anyone fetching from your local repository
should be aware that not all branches point to something sensible. If
they don't, they'll be sure to ask you so you can bring out the LART.

> This is all vaguely quilt-like, although I'd rather not worry about the
> order of patches.  I suppose I'd like git to let me "commit under" the
> final merge.  When I switch branches, git should reorganize the tree of
> merges so that the current branch is only one merge from the HEAD.
> 

Umm... Use a separate branch to do the merge, and enable rerere caching
so your recorded resolutions get reused when you do the merge "for real".

> (Another thing I've wished for that might be related is for a branch
> to have a notion of its origin.  So I can rebase it onto an arbitrary
> place in the commit tree without having to explicitly specify an origin.)
> 

This is a bad idea.

> ((Another really simple feature I keep wanting is "git commit -b
> <newbranch>".  I should probably try to write a patch...))
> 

This is a good idea (assuming you mean "commit this state to that branch",
and it should be fairly trivial to implement using something like this:

    git stash && git checkout -b newbranch && git stash pop && git commit

> 
> Anyway, my feature ideas might be unworkable, and in any case, they'll
> take a long time to implement.  Is there some easier way to achieve more
> or less this effect?
> 

I'm not sure what effect you're after. If you can write down, with very
*little* detail, what it is you want to achieve, it will be a lot easier
to help you than if you ask about small steps in a pre-thought-out solution
that may not even be right from the beginning.

> Maybe the planned git-rebase improvements to handle merges better will
> fix this, so I can just commit on top and periodically rebase the changes
> under the head manually without too much pain?  (git rebase -i -p does
> appear to be working better than I remember.)
> 

What planned rebase improvements are you talking about?

> H'm... in fact, it might be as easy as replacing "git pull" with
> git rebase -p -i <last merge>^
> (Delete the merge in the editor)
> git pull <remote>
> 
> Annoying to remember, but not TOO bad.

Use an alias, or a wrapper script.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: Git workflow: how to achieve?

[George Spelvin]

Andreas Ericsson <ae@op5.se> wrote:
>>  I want to be able to switch feature branches easily.
> 
> Well, git makes that absolutely trivial. No other vcs has as cheap
> branching as git does.

Yes, I'm quite aware, but I mean while keeping the same merged result.
That is, I'm compiling and testing base+feature_a+feature_b+feature_c,
but I want to be able to change which feature I'm committing to.

This comes up if I arrange some merges so that the "current branch" is
one below the top merge:

o--o--o--o--o--o--o <-- HEAD
           /  /  /
    ...c--c  /  /
      ...b--b  /
        ...a--a

And then want to switch to branch c.

> Sounds like you'd benefit somewhat from using TopGit or some other
> patch-queue management tool. I haven't used them myself, but I believe
> this is the kind of problem they set out to solve.

Thanks, I didn't know about TopGit; I'm looking at it now.

>> But when I want to be testing something highly volatile like linux-next,
>> and ensuring that my work continues to merge with it cleanly, as well
>> as helping others with their branches, it becomes a daily pain.
> 
> Why? Just merge it to make sure it merges, and then throw away the result.
> There's a post made by Linus somewhere on how he would like people to do
> merges (ie, which to keep and which to throw away).

I don't want to just merge it, I want to *compile it* and *run it*.
Because the whole point of using linux-next is to test it.
And when it breaks, I want access to the relevant source code.

>> The best attempt I have so far is to rebase a lot.  But that means that
>> I can't do any merging in my development branch lest the rebasing turn
>> into a mess.
> 
> Merging other branches into your development branch and keeping the result
> is often a bad idea, since you can't later merge the development branch
> without also merging everything that *you* merged. This makes it nearly
> impossible to pick your feature into random spots in the DAG. If you ever
> work anywhere but on the bleeding edge, you'll find merging the "main dev
> branch" is a *really* bad idea.

I didn't mean merging main into my dev branch, but merging within my
dev branch.  As in, "that refactorization of the foo_* helpers is a
better base for development, so I'll merge that into my bar branch", and
then test-merge THAT into main.

>>  And forcing everything to be linear makes changing branches
>> a pain.
> 
> Everything needn't be linear, but if you can manage to keep each topic
> linear that makes your series a lot easier to review.

Yes, I know.  I generally try, but it's occasionally nice to have sub-branches
within a feature branch.

>>  And I can't share half-finished versions with co-workers.
> 
> So finish it before you push. Anyone fetching from your local repository
> should be aware that not all branches point to something sensible. If
> they don't, they'll be sure to ask you so you can bring out the LART.
> 
>> This is all vaguely quilt-like, although I'd rather not worry about the
>> order of patches.  I suppose I'd like git to let me "commit under" the
>> final merge.  When I switch branches, git should reorganize the tree of
>> merges so that the current branch is only one merge from the HEAD.
> 
> Umm... Use a separate branch to do the merge, and enable rerere caching
> so your recorded resolutions get reused when you do the merge "for real".

>> (Another thing I've wished for that might be related is for a branch
>> to have a notion of its origin.  So I can rebase it onto an arbitrary
>> place in the commit tree without having to explicitly specify an origin.)
> 
> This is a bad idea.

Can you explain why?  When I talk about a "feature branch", I mean a chain
of commits that introduces a feature.  Why would it be bad for a reference
to have an optional "origin" attribute that indicates a start position?

>> ((Another really simple feature I keep wanting is "git commit -b
>> <newbranch>".  I should probably try to write a patch...))
> 
> This is a good idea (assuming you mean "commit this state to that branch",
> and it should be fairly trivial to implement using something like this:
> 
>     git stash && git checkout -b newbranch && git stash pop && git commit

Actually, just 
git checkout -b newbranch; git commit "$@"
does the job quite nicely, but 

>> Anyway, my feature ideas might be unworkable, and in any case, they'll
>> take a long time to implement.  Is there some easier way to achieve more
>> or less this effect?
> 
> I'm not sure what effect you're after. If you can write down, with very
> *little* detail, what it is you want to achieve, it will be a lot easier
> to help you than if you ask about small steps in a pre-thought-out solution
> that may not even be right from the beginning.

We may have a language problem.  "very little detail" means almost no detail,
an absence of details.  Did you mean "every little detail"?

It actually came to mind recently during $DAY_JOB work, but let me give a
concrete example based on the Linux kernel:

I am running a customized Linux kernel.  On top of Linus's base,
I have local patches to enable 64-bit DMA on the SB600 SATA controller,
some local patches to make the RAID (md) code print out the locations
of mismatches when verifying, the EDAC quilt series, a merge of the
LinuxPPS code, a number of local patches to the LinuxPPS code (that
I'm discussing with Rodolfo Giometti), and some revisions to the serial
interrupt handler (that I'm discussing with Alan Cox).

There's also the beginning of an ethernet driver that I'm trying to
write, but it's going slow.

Every week or two, I rebase all that on top of Linus's latest.  Plain
rebase doesn't like the LinuxPPS merge, so I've been doing it manually
in two parts.  Although rebase -p is apprently working much better than
I remember.

I would also like to be able to revise each of those features
independently, based on feedback from the relevant upstream maintainer.
I also don't know in what order they'll be acecpted upstream (if ever).
Regardless of that, I want to my running kernels to have all of the
patches applied.

Currently, some patches get deeply buried in the stack and I have to
do a lot of deep rebasing.

>> Maybe the planned git-rebase improvements to handle merges better will
>> fix this, so I can just commit on top and periodically rebase the changes
>> under the head manually without too much pain?  (git rebase -i -p does
>> appear to be working better than I remember.)
> 
> What planned rebase improvements are you talking about?

I was referring to Dscho's "Heads up: major rebase -i -p rework coming up"
http://marc.info/?l=git&m=123282871626215

>> H'm... in fact, it might be as easy as replacing "git pull" with
>> git rebase -p -i <last merge>^
>> (Delete the merge in the editor)
>> git pull <remote>
>> 
>> Annoying to remember, but not TOO bad.
> 
> Use an alias, or a wrapper script.

Once I've done it manually a few times to verify there are no surprises.

Re: Git workflow: how to achieve?

[Andreas Ericsson]

George Spelvin wrote:
> Andreas Ericsson <ae@op5.se> wrote:
>>>  I want to be able to switch feature branches easily.
>> Well, git makes that absolutely trivial. No other vcs has as cheap
>> branching as git does.
> 
> Yes, I'm quite aware, but I mean while keeping the same merged result.
> That is, I'm compiling and testing base+feature_a+feature_b+feature_c,
> but I want to be able to change which feature I'm committing to.
> 
> This comes up if I arrange some merges so that the "current branch" is
> one below the top merge:
> 
> o--o--o--o--o--o--o <-- HEAD
>            /  /  /
>     ...c--c  /  /
>       ...b--b  /
>         ...a--a
> 
> And then want to switch to branch c.
> 

Sorry, I still don't see a problem here. If you want to commit to C,
just checkout C and commit. What am I missing?

I know some kernel-lieutenants use a magic branch-prefix to mark which
branches to merge and compile-test, and then use a homegrown script
to actually merge all those branches.

> 
>>> But when I want to be testing something highly volatile like linux-next,
>>> and ensuring that my work continues to merge with it cleanly, as well
>>> as helping others with their branches, it becomes a daily pain.
>> Why? Just merge it to make sure it merges, and then throw away the result.
>> There's a post made by Linus somewhere on how he would like people to do
>> merges (ie, which to keep and which to throw away).
> 
> I don't want to just merge it, I want to *compile it* and *run it*.
> Because the whole point of using linux-next is to test it.
> And when it breaks, I want access to the relevant source code.
> 

Ok. Sounds like you'd need some CI tool with special smarts to try merging
all relevant branches and bailing out on conflicts or something, unless you
intend to compile it locally and run it from there.

> 
>>>  And I can't share half-finished versions with co-workers.
>> So finish it before you push. Anyone fetching from your local repository
>> should be aware that not all branches point to something sensible. If
>> they don't, they'll be sure to ask you so you can bring out the LART.
>>
>>> This is all vaguely quilt-like, although I'd rather not worry about the
>>> order of patches.  I suppose I'd like git to let me "commit under" the
>>> final merge.  When I switch branches, git should reorganize the tree of
>>> merges so that the current branch is only one merge from the HEAD.
>> Umm... Use a separate branch to do the merge, and enable rerere caching
>> so your recorded resolutions get reused when you do the merge "for real".
> 
>>> (Another thing I've wished for that might be related is for a branch
>>> to have a notion of its origin.  So I can rebase it onto an arbitrary
>>> place in the commit tree without having to explicitly specify an origin.)
>> This is a bad idea.
> 
> Can you explain why?  When I talk about a "feature branch", I mean a chain
> of commits that introduces a feature.  Why would it be bad for a reference
> to have an optional "origin" attribute that indicates a start position?
> 

Because they already have a reference to where they started. It's called
the merge-base and it differs depending on which branch you want to merge
it into. If you have 'maint', 'next' and 'master', the "point of origin"
for "next" is "maint" from the point of view of "maint", but it's "master"
from the point of view of "master". Marking it as "master" would mean you
couldn't use that mark when merging it into maint, so you'd have to either
mark it as a sub-branch of all previous branches, or do what's done today.

IOW, the mark would help one corner-case but would make all other cases
more complicated. A bad idea indeed.

>>> ((Another really simple feature I keep wanting is "git commit -b
>>> <newbranch>".  I should probably try to write a patch...))
>> This is a good idea (assuming you mean "commit this state to that branch",
>> and it should be fairly trivial to implement using something like this:
>>
>>     git stash && git checkout -b newbranch && git stash pop && git commit
> 
> Actually, just 
> git checkout -b newbranch; git commit "$@"
> does the job quite nicely, but 
> 
>>> Anyway, my feature ideas might be unworkable, and in any case, they'll
>>> take a long time to implement.  Is there some easier way to achieve more
>>> or less this effect?
>> I'm not sure what effect you're after. If you can write down, with very
>> *little* detail, what it is you want to achieve, it will be a lot easier
>> to help you than if you ask about small steps in a pre-thought-out solution
>> that may not even be right from the beginning.
> 
> We may have a language problem.  "very little detail" means almost no detail,
> an absence of details.  Did you mean "every little detail"?
> 

No, I mean with very little detail. Usually when trying to answer a question
and I can't make sense of the question itself it's because the person asking
has already started down some road and entered into a frame of mind which I
cannot share. That mindset may not be correct to solve the problem, so removing
detail usually helps.

"I'd like to automatically merge a bunch of branches and compile-test them
on every commit. How do I go about doing that?" is a much more open question
that invites to giving a lot more correct answers. Since you haven't asked
such a high-level question yet, I'm not sure what your need is, and therefore
I cannot help you find a suitable solution.

> It actually came to mind recently during $DAY_JOB work, but let me give a
> concrete example based on the Linux kernel:
> 
> I am running a customized Linux kernel.  On top of Linus's base,
> I have local patches to enable 64-bit DMA on the SB600 SATA controller,
> some local patches to make the RAID (md) code print out the locations
> of mismatches when verifying, the EDAC quilt series, a merge of the
> LinuxPPS code, a number of local patches to the LinuxPPS code (that
> I'm discussing with Rodolfo Giometti), and some revisions to the serial
> interrupt handler (that I'm discussing with Alan Cox).
> 
> There's also the beginning of an ethernet driver that I'm trying to
> write, but it's going slow.
> 
> Every week or two, I rebase all that on top of Linus's latest.  Plain
> rebase doesn't like the LinuxPPS merge, so I've been doing it manually
> in two parts.  Although rebase -p is apprently working much better than
> I remember.
> 

This step is rather unnecessary unless there are changes to API's you
depend on. Doing a single rebase once you're done with the patch-series
would be far better and would, I expect, remove quite a lot of the
complexity you're experiencing.

> I would also like to be able to revise each of those features
> independently, based on feedback from the relevant upstream maintainer.
> I also don't know in what order they'll be acecpted upstream (if ever).
> Regardless of that, I want to my running kernels to have all of the
> patches applied.
> 
> Currently, some patches get deeply buried in the stack and I have to
> do a lot of deep rebasing.
> 

I'm not sure what you mean by "deep rebasing" or "stack", unless you've
started using topgit already (which, I believe, does patch-stacks).

No patch should ever get "deeply buried" unless you do really, really
weird things with your topic-branches though. They *should* remain the
same each and every time.

But the "all patches in my running kernel" problem could pretty easily
be solved by simply merging all your features into one branch and then
periodically merge Linus' latest into that branch whenever you want to
update your kernel. When the merge conflicts, you'll have to figure out
which of your features cause it (shouldn't be a problem unless they
conflict with themselves) and then rebase *that* feature onto the new
upstream (or something).

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Register now for Nordic Meet on Nagios, June 3-4 in Stockholm
 http://nordicmeetonnagios.op5.org/

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: Git workflow: how to achieve?

[George Spelvin]

>From exon@op5.com Tue Apr 28 07:24:19 2009
Sender: Andreas Ericsson <exon@op5.com>
Date: Tue, 28 Apr 2009 09:24:02 +0200
From: Andreas Ericsson <ae@op5.se>
User-Agent: Thunderbird 2.0.0.21 (X11/20090320)
MIME-Version: 1.0
To: George Spelvin <linux@horizon.com>
CC: git@vger.kernel.org
Subject: Re: Git workflow: how to achieve?
References: <20090427225900.29793.qmail@science.horizon.com>
In-Reply-To: <20090427225900.29793.qmail@science.horizon.com>
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 7bit

Andreas Ericsson <ae@op5.se> wrote:
> George Spelvin wrote:
>> Andreas Ericsson <ae@op5.se> wrote:
>>>>  I want to be able to switch feature branches easily.
>>> Well, git makes that absolutely trivial. No other vcs has as cheap
>>> branching as git does.
>> 
>> Yes, I'm quite aware, but I mean while keeping the same merged result.
>> That is, I'm compiling and testing base+feature_a+feature_b+feature_c,
>> but I want to be able to change which feature I'm committing to.
>> 
>> This comes up if I arrange some merges so that the "current branch" is
>> one below the top merge:
>> 
>> o--o--o--o--o--o--o <-- HEAD
>>            /  /  /
>>     ...c--c  /  /
>>       ...b--b  /
>>         ...a--a
>> 
>> And then want to switch to branch c.

> Sorry, I still don't see a problem here. If you want to commit to C,
> just checkout C and commit. What am I missing?

If a or b touched one of the files that I edited (even in very different
places), the checkout will fail.  And even if it succeeds, I have to
re-do the merges.

I expect it would be easier to commit on top of the pile, test, and
then rebase the last few patches to push the new changes under the
merges.

I know some kernel-lieutenants use a magic branch-prefix to mark which
branches to merge and compile-test, and then use a homegrown script
to actually merge all those branches.

>>>> But when I want to be testing something highly volatile like linux-next,
>>>> and ensuring that my work continues to merge with it cleanly, as well
>>>> as helping others with their branches, it becomes a daily pain.
>>> Why? Just merge it to make sure it merges, and then throw away the result.
>>> There's a post made by Linus somewhere on how he would like people to do
>>> merges (ie, which to keep and which to throw away).
>> 
>> I don't want to just merge it, I want to *compile it* and *run it*.
>> Because the whole point of using linux-next is to test it.
>> And when it breaks, I want access to the relevant source code.
> 
> Ok. Sounds like you'd need some CI tool with special smarts to try merging
> all relevant branches and bailing out on conflicts or something, unless you
> intend to compile it locally and run it from there.

> Because they already have a reference to where they started. It's called
> the merge-base and it differs depending on which branch you want to merge
> it into. If you have 'maint', 'next' and 'master', the "point of origin"
> for "next" is "maint" from the point of view of "maint", but it's "master"
> from the point of view of "master". Marking it as "master" would mean you
> couldn't use that mark when merging it into maint, so you'd have to either
> mark it as a sub-branch of all previous branches, or do what's done today.
> 
> IOW, the mark would help one corner-case but would make all other cases
> more complicated. A bad idea indeed.

Er... obviously, merges have to work the way they do.  This is all for
rebases (cherry-picks).  And if the merge base is more recent than
the branch base, that should take precendence.

Basically, "git-rebase upstream branch" current cherry-picks the commits
in upstream..branch, a.k.a branch ^upstream.  I'd like it to be
branch ^branch.base ^upstream.

>> We may have a language problem.  "very little detail" means almost no detail,
>> an absence of details.  Did you mean "every little detail"?
> 
> No, I mean with very little detail. Usually when trying to answer a
> question and I can't make sense of the question itself it's because
> the person asking has already started down some road and entered into
> a frame of mind which I cannot share. That mindset may not be correct
> to solve the problem, so removing detail usually helps.

Ah, sorry, you wanted more abstract, not more concrete.

> "I'd like to automatically merge a bunch of branches and compile-test them
> on every commit. How do I go about doing that?" is a much more open question
> that invites to giving a lot more correct answers. Since you haven't asked
> such a high-level question yet, I'm not sure what your need is, and therefore
> I cannot help you find a suitable solution.

Okay, "I'd like to commit to somewhere further back in the history rather
than HEAD, and have the HEAD automagically rebased".  How do I do that?

>> It actually came to mind recently during $DAY_JOB work, but let me give a
>> concrete example based on the Linux kernel:
>> 
>> I am running a customized Linux kernel.  On top of Linus's base,
>> I have local patches to enable 64-bit DMA on the SB600 SATA controller,
>> some local patches to make the RAID (md) code print out the locations
>> of mismatches when verifying, the EDAC quilt series, a merge of the
>> LinuxPPS code, a number of local patches to the LinuxPPS code (that
>> I'm discussing with Rodolfo Giometti), and some revisions to the serial
>> interrupt handler (that I'm discussing with Alan Cox).
>> 
>> There's also the beginning of an ethernet driver that I'm trying to
>> write, but it's going slow.
>> 
>> Every week or two, I rebase all that on top of Linus's latest.  Plain
>> rebase doesn't like the LinuxPPS merge, so I've been doing it manually
>> in two parts.  Although rebase -p is apprently working much better than
>> I remember.
> 
> This step is rather unnecessary unless there are changes to API's you
> depend on. Doing a single rebase once you're done with the patch-series
> would be far better and would, I expect, remove quite a lot of the
> complexity you're experiencing.

*Frustration*  I'm somehow having trouble communicating.

I need to rebase it so I have a merged source tree that I can compile
and test.  How else can I test Linus's latest plus my local changes?
What alternative are you suggesting?

I want, AT ALL TIMES, to be running the kernel consisting of
Linus's latest plus my various local hacks.  I want to be able to
freely update any of the components that make up the result,
and have the sum automatically recomputed for me.

>> Currently, some patches get deeply buried in the stack and I have to
>> do a lot of deep rebasing.
> 
> I'm not sure what you mean by "deep rebasing" or "stack", unless you've
> started using topgit already (which, I believe, does patch-stacks).
> 
> No patch should ever get "deeply buried" unless you do really, really
> weird things with your topic-branches though. They *should* remain the
> same each and every time.

I mean that the patch I want to edit is 20 or more patches back
in history (I'm running 2.6.30-rc3-00063-gd3de9aa at the moment),
so amending it involves a considerable amount of rebasing.

(Because I'm currently organized as a linear list of local
patches on top of upstream.  I'd prefer separate feature
branches plus merges, but that's what I'm asking how to make
work efficiently.)

Re: Git workflow: how to achieve?

[Andreas Ericsson]

George Spelvin wrote:
>>> We may have a language problem.  "very little detail" means almost no detail,
>>> an absence of details.  Did you mean "every little detail"?
>> No, I mean with very little detail. Usually when trying to answer a
>> question and I can't make sense of the question itself it's because
>> the person asking has already started down some road and entered into
>> a frame of mind which I cannot share. That mindset may not be correct
>> to solve the problem, so removing detail usually helps.
> 
> Ah, sorry, you wanted more abstract, not more concrete.
> 
>> "I'd like to automatically merge a bunch of branches and compile-test them
>> on every commit. How do I go about doing that?" is a much more open question
>> that invites to giving a lot more correct answers. Since you haven't asked
>> such a high-level question yet, I'm not sure what your need is, and therefore
>> I cannot help you find a suitable solution.
> 
> Okay, "I'd like to commit to somewhere further back in the history rather
> than HEAD, and have the HEAD automagically rebased".  How do I do that?
> 

Now we're talking.

If I were you, I'd do (assuming 'master' is upstream's head):

git checkout -b private/test master
git merge $all_feature_branches
# some sort of failure detected
git checkout $failed_topic
git rebase -i -p $(git merge-base HEAD master)
# select 'edit' for the commits you want to edit
git branch -D private/test
# goto 1


There may be even simpler workflows. I do not know.

>>> It actually came to mind recently during $DAY_JOB work, but let me give a
>>> concrete example based on the Linux kernel:
>>>
>>> I am running a customized Linux kernel.  On top of Linus's base,
>>> I have local patches to enable 64-bit DMA on the SB600 SATA controller,
>>> some local patches to make the RAID (md) code print out the locations
>>> of mismatches when verifying, the EDAC quilt series, a merge of the
>>> LinuxPPS code, a number of local patches to the LinuxPPS code (that
>>> I'm discussing with Rodolfo Giometti), and some revisions to the serial
>>> interrupt handler (that I'm discussing with Alan Cox).
>>>
>>> There's also the beginning of an ethernet driver that I'm trying to
>>> write, but it's going slow.
>>>
>>> Every week or two, I rebase all that on top of Linus's latest.  Plain
>>> rebase doesn't like the LinuxPPS merge, so I've been doing it manually
>>> in two parts.  Although rebase -p is apprently working much better than
>>> I remember.
>> This step is rather unnecessary unless there are changes to API's you
>> depend on. Doing a single rebase once you're done with the patch-series
>> would be far better and would, I expect, remove quite a lot of the
>> complexity you're experiencing.
> 
> *Frustration*  I'm somehow having trouble communicating.
> 
> I need to rebase it so I have a merged source tree that I can compile
> and test.  How else can I test Linus's latest plus my local changes?
> What alternative are you suggesting?
> 
> I want, AT ALL TIMES, to be running the kernel consisting of
> Linus's latest plus my various local hacks.  I want to be able to
> freely update any of the components that make up the result,
> and have the sum automatically recomputed for me.
> 

Just merge them in. You're going through an obscene amount of trouble
just to keep everything linear when there's no need for you to do so.

  git checkout private/test
  git merge master

to merge latest upstream stuff. If you get conflicts, you may have to
amend your patch-series with one or more extra commits to resolve those
conflicts, but there's no need to rebase them again if there are zero
conflicts (and all tests pass, ofcourse; Not all post-merge code
conflicts are caused by the actual merge algorithm).

If there are no new breakages or no new conflicts, there's no need to
rebase your topics again at this point.

>>> Currently, some patches get deeply buried in the stack and I have to
>>> do a lot of deep rebasing.
>> I'm not sure what you mean by "deep rebasing" or "stack", unless you've
>> started using topgit already (which, I believe, does patch-stacks).
>>
>> No patch should ever get "deeply buried" unless you do really, really
>> weird things with your topic-branches though. They *should* remain the
>> same each and every time.
> 
> I mean that the patch I want to edit is 20 or more patches back
> in history (I'm running 2.6.30-rc3-00063-gd3de9aa at the moment),
> so amending it involves a considerable amount of rebasing.
> 
> (Because I'm currently organized as a linear list of local
> patches on top of upstream.  I'd prefer separate feature
> branches plus merges, but that's what I'm asking how to make
> work efficiently.)

I think I answered this up above, but I'm superbly tired right now so
it's entirely possible I misunderstood one thing or another.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Register now for Nordic Meet on Nagios, June 3-4 in Stockholm
 http://nordicmeetonnagios.op5.org/

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: Git workflow: how to achieve?

[Sitaram Chamarty]

On 2009-04-28, George Spelvin <linux@horizon.com> wrote:

> I want, AT ALL TIMES, to be running the kernel consisting of
> Linus's latest plus my various local hacks.  I want to be able to
> freely update any of the components that make up the result,
> and have the sum automatically recomputed for me.

[and]

> (Because I'm currently organized as a linear list of local
> patches on top of upstream.  I'd prefer separate feature
> branches plus merges, but that's what I'm asking how to make
> work efficiently.)

I'll freely admit I'm focusing on just those two paras out
of a fairly long email, but topgit does precisely this.
I've started to use it, and it seems to do the job well.
When a feature branch gets updated, you just check out what
I will call "t/all" and do a "tg update" and it picks up the
new stuff.

So the everyday stuff works.

The only sort-of-open question, right now, is how do you
remove a dependency when one of your "local hacks" grows up
and goes off to college.  Er... I mean gets accepted
upstream :-)

My guess is that as long as all these local hacks are indeed
local, the branch that represents the "sum total" can be
blown away and rebuilt when that happens; you'd just list
all but the recently graduated feature as dependencies.

Regards,

Sitaram