Cherry picking instead of merges.

Cherry picking instead of merges.

[David Brown]

We have a branch coming from "company B" who is also doing development on
the same tree as us.  We would like to do regular merges with them, but
unfortunately, there are quite a few changes in between.

First we tried a git-merge and resolved the conflicts.  The problem here is
that the resultant code didn't work.  git-bisect wasn't very useful because
the intermediate versions don't have resolved conflicts.

Yesterday, one developer cherry picked company B's changes into a branch.
It appears he resolved the conflicts for each commit, which should make
bisecting easier.

The problem is that we now have very divergent history.

Any advice on how to make use of how he resolved conflicts in order to
merge company B's changes in using git-merge.  I could always use the
"ours" strategy to just kind of force the merge, so at least future merges
would work, but it'd kind of be nice to have the proper history.

There are about 110 commits in question.

Thanks,
David Brown

Re: Cherry picking instead of merges.

[Alex Riesen]

David Brown, Thu, Jul 03, 2008 20:26:50 +0200:
> Yesterday, one developer cherry picked company B's changes into a branch.
> It appears he resolved the conflicts for each commit, which should make
> bisecting easier.
>
> The problem is that we now have very divergent history.

...and have absolutely no idea on what were the changes of the company
B were made.

> Any advice on how to make use of how he resolved conflicts in order to
> merge company B's changes in using git-merge. ...

Let that developer do the merge next time? Or let the B do a merge
with a commonly accepted base?

Re: Cherry picking instead of merges.

[Avery Pennarun]

On 7/3/08, David Brown <git@davidb.org> wrote:
>  First we tried a git-merge and resolved the conflicts.  The problem here is
>  that the resultant code didn't work.  git-bisect wasn't very useful because
>  the intermediate versions don't have resolved conflicts.
>
>  Yesterday, one developer cherry picked company B's changes into a branch.
>  It appears he resolved the conflicts for each commit, which should make
>  bisecting easier.
>
>  The problem is that we now have very divergent history.
>
>  Any advice on how to make use of how he resolved conflicts in order to
>  merge company B's changes in using git-merge.  [...]

Unfortunately, since your mismerged branches are already published,
rewriting history would cause a lot of pain for everyone.  It would be
better to avoid doing that entirely.  However, I can see why you'd
want to do that in order to make future git-bisect easier.

Basically, if you're going to try to fix the git-bisect intermediate
versions, you're going to have to rewrite history anyway; in which
case, why not just make your developer's cherry-picked branch the
official one?  Then your problems are solved, other than getting all
your developers onto the new history.

Alternatively, if you just want to fix your main development tree so
that it's "correct", then you could do this:

- Go to a point in time where both branches (main and developer) have
exactly the same set of patches: that is, your company + company B.
The exact history (ie. commit ids) will look different, because the
two branches took different paths to get there, but the code *should*
have been identical at those two times, since you have the same set of
patches.  Call the two points X and Y.

- X (on the main branch) is actually different from Y (on your
developer's branch) because someone mis-resolved the conflicts on X.
However, the only difference between X and Y should be the mis-merge.
Thus:

- git diff X..Y >fix-mis-merge.patch

- git checkout HEAD

- git apply fix-mis-merge.patch

This will apply the correct conflict resolution to the tip of your
newest branch.  All the revisions between X and HEAD will still be
broken, but that's usually better than trying to rewrite history and
pretend the broken revisions never existed.  You can always use "git
bisect skip" for cases like that.

Have fun,

Avery

Re: Cherry picking instead of merges.

[David Brown]

On Thu, Jul 03, 2008 at 04:15:22PM -0400, Avery Pennarun wrote:

>Unfortunately, since your mismerged branches are already published,
>rewriting history would cause a lot of pain for everyone.  It would be
>better to avoid doing that entirely.  However, I can see why you'd
>want to do that in order to make future git-bisect easier.

It's only sitting in a private developer's branch.  I want to do the merge
properly, but I'm just trying to figure out how to get the conflict
resolution out of his work.

>Basically, if you're going to try to fix the git-bisect intermediate
>versions, you're going to have to rewrite history anyway; in which
>case, why not just make your developer's cherry-picked branch the
>official one?  Then your problems are solved, other than getting all
>your developers onto the new history.

Once we start cherry picking the changes from Company B, we have a
different set of changes from them, and future merges will get harder and
harder.

>This will apply the correct conflict resolution to the tip of your
>newest branch.  All the revisions between X and HEAD will still be
>broken, but that's usually better than trying to rewrite history and
>pretend the broken revisions never existed.  You can always use "git
>bisect skip" for cases like that.

Except we already know that the broken change is inside of the broken
revisions.

It turns out that things are more messed up than I thought.  This developer
had done a 'git push' with some manual refs and pushed what was supposed to
be a merge into an unmerged branch.

I've spoken with all of the developers who use this tree, and everyone
agrees that rewinding the tree is the best way to go.  Now just time to
trudge forward and learn.

But, it makes for a good new rule: no cherry-picking other people's
changes.

David

Re: Cherry picking instead of merges.

[Samuel Tardieu]

>>>>> "David" == David Brown <git@davidb.org> writes:

David> It's only sitting in a private developer's branch.  I want to
David> do the merge properly, but I'm just trying to figure out how to
David> get the conflict resolution out of his work.

Why don't you create a commit which has both your (A) and company B
branch head (B) as parent with the same content as the private
developer's branch (P) which contains the succesful merge?

Something like that: (totally untested)

% git checkout A
% git merge --no-commit -s ours B
% git checkout P .
% git commit -a

Re: Cherry picking instead of merges.

[Linus Torvalds]

On Thu, 3 Jul 2008, David Brown wrote:
> 
> First we tried a git-merge and resolved the conflicts.  The problem here is
> that the resultant code didn't work.  git-bisect wasn't very useful because
> the intermediate versions don't have resolved conflicts.
> 
> Yesterday, one developer cherry picked company B's changes into a branch.
> It appears he resolved the conflicts for each commit, which should make
> bisecting easier.

What I would suggest is actually a stupid combination of the two. The 
advantage and disadvantage of the cherry-picking is:

 - the cherry-picking obviously gave you the end result you wanted, and is 
   bisectable at a per-commit level (ie each cherry-pick was a 
   "micro-merge", and if there was a merge problem, it would show up 
   exactly in _that_ one)

BUT

 - the cherry-picking doesn't help future merges, since the history is 
   disjoint, and as such it will always diverge and just cause more and 
   more problems down the road to keep on doing this.

so what I would suggest is

 - generate the *state* of the tree by cherry-picking each commit one by 
   one, and verify that state.

 - but then do a merge that picks that one state as the merge result 
   (this is all assuming you cherry-picked every single commit, of 
   course!)

The latter thing is really easy to do. Just do the merge, and ignore the 
conflicts that happen entirely, because you then just use the tree from 
the temporary cherry-picking branch

	# Try to merge the other branch ..
	git merge otherbranch

	# .. but ignore conflicts entirely by then
	# setting the index and checked-out state to
	# the known-good end result
	git read-tree -u --reset cherry-picked-branch

	# .. and commit that instead
	git commit

	# and you can now remove the cherry-picked branch
	# that was only used for the tree
	git branch -D cherry-picked-branch

to actually commit that merge and get rid of the temporary cherry-picking 
branch.

(Of course, if the merge actually doesn't generate any conflicts, and thus 
commits it, but doesn't actually _work_, then that "git commit" needs to 
be a "git commit --amend" instead, in order to actually replace the merge 
commit rather than add a fixup commit afterwards)

End result: you have a nice merge with nice history that actually 
converges at a common point, but you effectively did the merge resolution 
one commit at a time with cherry-picking (or "git rebase", which is 
obviously just a convenient shorthand for cherry-picking everything).

This actually has a few other advantages too: "git show --cc" on that 
merge will also show the conflicts the way they got resolved, so you 
actually end up with some useful information this way too.

			Linus

Re: Cherry picking instead of merges.

[David Brown]

On Thu, Jul 03, 2008 at 02:18:53PM -0700, Linus Torvalds wrote:

>End result: you have a nice merge with nice history that actually 
>converges at a common point, but you effectively did the merge resolution 
>one commit at a time with cherry-picking (or "git rebase", which is 
>obviously just a convenient shorthand for cherry-picking everything).

I'm still not clear how the one-commit-at-a-time resolution gets recorded
anywhere (except in the cherry-picking branch).

It seems to be that I would need to do multiple merges, one at each point
where there is a conflict that I had to resolved.  I would remember this as
I did each cherry picked change, but after the fact, I would have to
compare the cherry picked change with the one it came from, and figure out
where conflicts had to be resolved.

Thanks,
David

Re: Cherry picking instead of merges.

[Björn Steinbrink]

On 2008.07.03 15:39:49 -0700, David Brown wrote:
> On Thu, Jul 03, 2008 at 02:18:53PM -0700, Linus Torvalds wrote:
>
>> End result: you have a nice merge with nice history that actually
>> converges at a common point, but you effectively did the merge
>> resolution one commit at a time with cherry-picking (or "git rebase",
>> which is obviously just a convenient shorthand for cherry-picking
>> everything).
>
> I'm still not clear how the one-commit-at-a-time resolution gets
> recorded anywhere (except in the cherry-picking branch).

They don't get recorded. Git does not store how merges happened in some
diffs or whatever, it just stores the end result. So what Linus
suggested was to reuse the end result that you already have to create
the merge commit.

Say you have 5 commits like this:

  A---B---C
   \
    D---E

Now there are two ways to get the changes from D and E ontop of C. The
first one is cherry-picking, which leads to:

  A---B---C---D'---E'
   \
    D---E

The other one is merging:

  A---B---C---M
   \         /
    D---E---/

Of course, you should end up with the same tree either way. It's just a
different way of getting towards that final state. So commit E' and
commit M, while different commits, would point to the same tree object.

Now if you want to create that merge commit M, and already have E' (like
you do, as you already did all those cherry-picks), you can use what
Linus suggested. You start the merge, ignore any conflicts and just tell
git to use the tree that E' is pointing to instead.

Björn

Re: Cherry picking instead of merges.

[David Brown]

On Fri, Jul 04, 2008 at 02:10:03AM +0200, Björn Steinbrink wrote:

>The other one is merging:
>
>  A---B---C---M
>   \         /
>    D---E---/
>
>Of course, you should end up with the same tree either way. It's just a
>different way of getting towards that final state. So commit E' and
>commit M, while different commits, would point to the same tree object.

Ok.  I guess I need to explain a little further.  The advice I've gotten is
good, BTW.

After we've resolved the merge and committed it, we've then discovered that
it doesn't work.  However, there are around 100 commits on both branches of
the merges and it would be nice to come up with some way of doing something
like a bisect.

The trick is that the branch being merged in doesn't actually work on our
platform, so I can't just test the alternate branch.

But, git-bisect isn't being all that helpful here.  The problem is that the
only conflicts we resolved is how the two trees were put together.  Picking
points in the middle seem to generate lots of similar, but not quite the
same conflicts.

A cherry-picked tree would allow for an easy bisect, since all of the
intermediary versions would work.  If I somehow knew magical points within
the other tree I could do some number of merges and the bisect would still
work.  I suppose I could do the merges one at a time, but it would make the
history rather verbose.

Thanks,
David

Re: Cherry picking instead of merges.

[Linus Torvalds]

On Thu, 3 Jul 2008, David Brown wrote:
> 
> A cherry-picked tree would allow for an easy bisect, since all of the
> intermediary versions would work.  If I somehow knew magical points within
> the other tree I could do some number of merges and the bisect would still
> work.  I suppose I could do the merges one at a time, but it would make the
> history rather verbose.

Why do you want to worry so much about bisecting the merge?

You should basically expect that normally, a merge doesn't cause new bugs 
in itself. And if it does, the most common case is a fairly obvious 
mismerge (we've had a number of those in the kernel, but they weren't all 
that mysterious). So _planning_ for a subtle bisection failure sounds a 
bit strange and counter-productive.

So don't plan for it, and don't make it a primary issue for your workflow. 

Then, *after* you have a merge, and the unlucky thing happens where the 
merge is actually the thing that introduces the bug, and it's not obvious 
what the exact interaction is, and bisection didn't help you pinpoint it 
(because it really was the merge), *that* is when you might then choose to 
re-do the rebase in a temporary branch, and then bisect that re-linearized 
history. 

So basically:

 - keep the history clean and obvious

 - make it fairly straightforward and don't worry about the unlikely case.

 - don't do something painful just because bad things _can_ happen (they 
   can happen regardless of what you do)

 - because in the unlikely situation that bad things happen, you can 
   always then go back and do things the other way around and pinpoint 
   where the fault is that way.

IOW, let's say that you really do bisect things down to a merge and cannot 
see what the fault in that merge is, you can literally do

	# create a test-branch with the 'remote' side of the merge
	git checkout -b test-branch merge^2

	# rebase that remote side on top of the local side
	git rebase merge^

and now you've linearized the merge temporarily just to be able to bisect 
in that temporary branch what the bad interaction is. But once you've 
bisected it, the temporary branch is again just junk - there's no real 
value in saving it, because once you know _why_ the bug happened, you're 
just better off going back to the original history and just fixing it (and 
documenting the bug through the fix, rather than by addign extra-ugly 
history).

		Linus

Re: Cherry picking instead of merges.

[Johannes Sixt]

Linus Torvalds schrieb:
> IOW, let's say that you really do bisect things down to a merge and cannot 
> see what the fault in that merge is, you can literally do
> 
> 	# create a test-branch with the 'remote' side of the merge
> 	git checkout -b test-branch merge^2
> 
> 	# rebase that remote side on top of the local side
> 	git rebase merge^
> 
> and now you've linearized the merge temporarily just to be able to bisect 
> in that temporary branch what the bad interaction is. But once you've 
> bisected it, the temporary branch is again just junk - there's no real 
> value in saving it, because once you know _why_ the bug happened, you're 
> just better off going back to the original history and just fixing it (and 
> documenting the bug through the fix, rather than by addign extra-ugly 
> history).

FWIW, the same thing in different words is written in section

"Why bisecting merge commits can be harder than bisecting linear history"

of Documentation/user-manual.txt.

-- Hannes

Re: Cherry picking instead of merges.

[Linus Torvalds]

On Fri, 4 Jul 2008, Johannes Sixt wrote:
> 
> FWIW, the same thing in different words is written in section
> 
> "Why bisecting merge commits can be harder than bisecting linear history"
> 
> of Documentation/user-manual.txt.

I don't think that's the same thign at all.

That section basically says "just keep things linear". Which I very much 
disagree with. Trying to keep things linear just doesn't work if you work 
together with other people - since you have to rewrite history.

So my argument was the exact _reverse_. Don't try to keep things linear, 
because it doesn't _work_ right. Do the merges. They will very seldom 
cause subtle merge problems (non-subtle ones are much more common, but 
trivial to handle), and they will mean that you can work effectively 
togethr with other people.

And then, _if_ you have a merge that you really cannot figure out why it 
breaks, at that point you can _temporarily_ linearize the git history 
after-the-fact just as easily as you would ever have done before-the-fact.

In other words: linearization throws away real and useful information. You 
can always linearize afterwards for bisection purposes or whatever, but 
you can never _un_linearize because you've thrown away the information.

So it's much better to just do merges and keep the history, and then there 
are ways to rewrite the history later if you really need it.

That said - I think it's good practice and perfectly sane to do things 
like git-rebase to rewrite the history in a _private_ tree that contains 
only your own modifications and has never been public (where "applied 
emailed patches from others" still counts as your own work).

The "don't linearize" mantra really only is about commits that have ever 
been in anybody elses repository (and whether they _came_ from there, or 
whether they came from you but were public to others is immaterial).

			Linus

Re: Cherry picking instead of merges.

[Linus Torvalds]

On Thu, 3 Jul 2008, David Brown wrote:
> 
> I'm still not clear how the one-commit-at-a-time resolution gets recorded
> anywhere (except in the cherry-picking branch).

For merges, the _only_ thing that matters from a "resolution" standpoint 
is that the history joins together. And it doesn't matter if it joins 
together fifty times or once - only the last point is relevant, since that 
becomes the last point of shared state.

> It seems to be that I would need to do multiple merges, one at each point
> where there is a conflict that I had to resolved.  I would remember this as
> I did each cherry picked change, but after the fact, I would have to
> compare the cherry picked change with the one it came from, and figure out
> where conflicts had to be resolved.

No, because you simply don't care. You only care about the final result. 
With a single merge, there is only a single merge-point, and a single 
result. The advantage of doing the cherry-pick is just that it splits the 
decision on how to merge up into many smaller decisions (that is often the 
_dis_advantage of cherry-picking too - it often makes for more work, even 
if the individual work may be simpler).

But once you've done all the small cherry-pick decisions, you'd only make 
one final merge that takes that final result as-is. Any future work will 
then know about the fact that you have a new common base point.

		Linus