Dealing with an upstream cherry-picked branch

Dealing with an upstream cherry-picked branch

[Jay Soffian]

I have the following scenario:

  o---o---Ma---o---o    local-master
 /       /
|       | .-b'------d'  upstream-a
|       |/  :       :
o---o---a---b---c---d   upstream-master

Local-master branched from upstream-master in distant past.
upstream-master periodically cuts tentative release branch upstream-a.
When they do this, that branch point (a) is merged into local-master
(Ma).

Over time, upstream applies fixes to upstream-a, but does so by
committing the fixes to upstream-master and then cherry-picking them
to upstream-a.

The question is how to best integrate the fixes on upstream-a into
local-master, w/o causing a headache when upstream cuts the next
tentative release branch, at which point upstrea-master will again
need to be merged into local-master (and that will also have other
local development). Here are two options I've considered:

1) Create a local-a integration branch, merged from upstream-a and
local-master. Keep this branch up-to-date by periodically merging
local-master and upstream-a:

  o---o---Ma---o---o     local-master
 /       /  \       \
|       |    `o------`o  local-a
|       |    /       /
|       | .-b'------d'   upstream-a
|       |/  :       :
o---o---a---b---c---d    upstream-master

2) Periodically merge upstream-a into local-master:

  o---o---Ma--o---o---o  local-master
 /       /   /       /
|       | .-b'------d'   upstream-a
|       |/  :       :
o---o---a---b---c---d    upstream-master

Then when it is next time to merge upstream-master into local-master either:

(a) Backout the upstream-a merges to local-master by reverting the
merge commits which introduced them to local-master, then merge
upstream-master.

(b) Just merge upstream-master and carefully deal with all the
conflicts. I think this will necessarily be an evil merge.

(c) Create a new branch at point Ma and cherry-pick only the local
commits from local-master past point Ma. This essentially gives me the
clean local-master I would've had if I'd been doing (1) all along.

- Are there any other options I'm missing?

- If I'm going to do (2a), I'm wondering if I'm missing any
subtleties. I've read the revert-a-faulty-merge how-to and I realize
my history won't be the cleanest, but I think it should work and leave
a picture like:

  o---o---Ma--o---o---o---o---Wd'---Wb'---Mi  local-master
 /       /   /       /                   /
|       | .-b'------d' upstream-a       |
|       |/  :       :                   |
o---o---a---b---c---d---e---f---g---h---i    upstream-master

Wd' is the revert of d' into local master. Wb' is the revert of b'
into local master. This reverts both merges from upstream-a into local
master. There may be conflicts to resolve due to the local changes
that happened in local-master. However, local-master should now be
"clean" to merge in upstream-master w/o having to worry about
conflicts between b and b', d and d'. Correct?

Thanks,

j.

Re: Dealing with an upstream cherry-picked branch

[Avery Pennarun]

On Sun, Mar 14, 2010 at 11:17 PM, Jay Soffian <jaysoffian@gmail.com> wrote:
> The question is how to best integrate the fixes on upstream-a into
> local-master, w/o causing a headache when upstream cuts the next
> tentative release branch, at which point upstrea-master will again
> need to be merged into local-master (and that will also have other
> local development). Here are two options I've considered:

You forgot option 0: tell upstream not to do that.

0a) Have them do their bugfixes directly in the upstream-a branch,
then merge sometimes from upstream-a to upstream-master.  Then when
they cut upstream-b from upstream-master, it should be an easy merge
for you (since they've already resolved any conflicts between
upstream-a and upstream-master as they arose over time).

0b) If they really need to do their bugfixes on upstream-master and
then cherry-pick them back to upstream-a, have them merge upstream-a
into upstream-master sometimes *anyway*, resolving any (probably
trivial) conflicts as they arise.  I say the conflicts should be
trivial because they're just cherry-picks anyway, so git will mostly
notice they're identical and ignore them.  This is easy to do a little
at a time, but gets more complicated if you wait a long time between
merges.

0c) Like 0b), only use 'git merge -s ours' to merge from upstream-a
into upstream-master.  This assumes that upstream-a *never* has a fix
other than one that is already in upstream-master, so that all
conflicts necessarily resolve to exactly what's already in
upstream-master.  This makes it easier for downstream people to merge
but doesn't actually cost any extra effort.

As for your original suggestions:

> 1) Create a local-a integration branch, merged from upstream-a and
> local-master. Keep this branch up-to-date by periodically merging
> local-master and upstream-a:

This will be inconvenient since local-master won't actually be useful;
if upstream-a contains a critical patch, you won't be able to test
your changes in local-master until you merge them into local-a.  Thus
the history of local-master, while "clean", will actually be
meaningless.

> 2) Periodically merge upstream-a into local-master:
> [...]
> Then when it is next time to merge upstream-master into local-master either:
>
> (a) Backout the upstream-a merges to local-master by reverting the
> merge commits which introduced them to local-master, then merge
> upstream-master.

Don't try to revert merge commits; that generally ends in disaster,
both in terms of your tree's correctness and your ability to
accurately retrace the history of your branch.  Possibly you can make
it work, but I don't know anybody who has.  Even if you can, you'll
still hate yourself in the morning.

> (b) Just merge upstream-master and carefully deal with all the
> conflicts. I think this will necessarily be an evil merge.

This is probably not as hard as it sounds, particularly if upstream-a
is *purely* a subset (in terms of cherry-picks, not in terms of
history) of upstream-master.  I'd recommend something like this (just
once when it's time to move to a new release branch):

       git checkout -b mergey upstream-master
       # take the history of upstream-a but not the content:
       git merge -s ours upstream-a
       # assuming local-master is your branch based on upstream-a:
       git checkout -b local-b local-master
       # merge the changes from upstream-a to upstream-master into local-b:
       git merge mergey

> (c) Create a new branch at point Ma and cherry-pick only the local
> commits from local-master past point Ma. This essentially gives me the
> clean local-master I would've had if I'd been doing (1) all along.

This will work, but the short form for exactly the same operation is:

        git checkout -b local-b local-master
        git rebase --onto upstream-master upstream-a

It will result in a cleaner history, and importantly, one that
upstream would probably be willing to merge someday.  You won't have
any extra cherry-picked commits confusingly merged into your history
with the original commits.

The disadvantage is that each of your branches will have a disjoint
history: there will be no branch showing how you got from local-a to
local-b, since they're actually two totally different things.  In git,
the ideal case is that if you look at the history of HEAD, you can see
the *entire* evolution of the product, and using a different
merge-base for each branch gets in the way of that.

The best option above, IMHO, is my proposed option 0.  But any of the
others will work.

Have fun,

Avery

Re: Dealing with an upstream cherry-picked branch

[Jay Soffian]

On Mon, Mar 15, 2010 at 2:46 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
> You forgot option 0: tell upstream not to do that.

I have no control over upstream and its currently policy absolutely is
not going to change.

However, I do have control of my mirror of upstream so...

> 0a) Have them do their bugfixes directly in the upstream-a branch,
> then merge sometimes from upstream-a to upstream-master.  Then when
> they cut upstream-b from upstream-master, it should be an easy merge
> for you (since they've already resolved any conflicts between
> upstream-a and upstream-master as they arose over time).
>
> 0b) If they really need to do their bugfixes on upstream-master and
> then cherry-pick them back to upstream-a, have them merge upstream-a
> into upstream-master sometimes *anyway*, resolving any (probably
> trivial) conflicts as they arise.  I say the conflicts should be
> trivial because they're just cherry-picks anyway, so git will mostly
> notice they're identical and ignore them.  This is easy to do a little
> at a time, but gets more complicated if you wait a long time between
> merges.
>
> 0c) Like 0b), only use 'git merge -s ours' to merge from upstream-a
> into upstream-master.  This assumes that upstream-a *never* has a fix
> other than one that is already in upstream-master, so that all
> conflicts necessarily resolve to exactly what's already in
> upstream-master.  This makes it easier for downstream people to merge
> but doesn't actually cost any extra effort.

... I can locally perform the 0b or 0c that they will not do. They
never have a fix not already in upstream-master, so I can do 0c.
[EDIT: I see this is what you suggest below.] I can also put a sanity
check in place to make sure there isn't something unexpected in
upstream-a.

I will explore this option.

> As for your original suggestions:
>
>> 1) Create a local-a integration branch, merged from upstream-a and
>> local-master. Keep this branch up-to-date by periodically merging
>> local-master and upstream-a:
>
> This will be inconvenient since local-master won't actually be useful;
> if upstream-a contains a critical patch, you won't be able to test
> your changes in local-master until you merge them into local-a.  Thus
> the history of local-master, while "clean", will actually be
> meaningless.

Okay.

>> 2) Periodically merge upstream-a into local-master:
>> [...]
>> Then when it is next time to merge upstream-master into local-master either:
>>
>> (a) Backout the upstream-a merges to local-master by reverting the
>> merge commits which introduced them to local-master, then merge
>> upstream-master.
>
> Don't try to revert merge commits; that generally ends in disaster,
> both in terms of your tree's correctness and your ability to
> accurately retrace the history of your branch.  Possibly you can make
> it work, but I don't know anybody who has.  Even if you can, you'll
> still hate yourself in the morning.

Okay, I'll take this as sage advice. :-)

>> (b) Just merge upstream-master and carefully deal with all the
>> conflicts. I think this will necessarily be an evil merge.
>
> This is probably not as hard as it sounds, particularly if upstream-a
> is *purely* a subset (in terms of cherry-picks, not in terms of
> history) of upstream-master.  I'd recommend something like this (just
> once when it's time to move to a new release branch):
>
>       git checkout -b mergey upstream-master
>       # take the history of upstream-a but not the content:
>       git merge -s ours upstream-a
>       # assuming local-master is your branch based on upstream-a:
>       git checkout -b local-b local-master
>       # merge the changes from upstream-a to upstream-master into local-b:
>       git merge mergey
>
>> (c) Create a new branch at point Ma and cherry-pick only the local
>> commits from local-master past point Ma. This essentially gives me the
>> clean local-master I would've had if I'd been doing (1) all along.
>
> This will work, but the short form for exactly the same operation is:
>
>        git checkout -b local-b local-master
>        git rebase --onto upstream-master upstream-a

Are you sure that will skip the merges into local-master from
upstream-a? I didn't think so and it's not my recollection based on
using --onto previously.

> It will result in a cleaner history, and importantly, one that
> upstream would probably be willing to merge someday.  You won't have
> any extra cherry-picked commits confusingly merged into your history
> with the original commits.
>
> The disadvantage is that each of your branches will have a disjoint
> history: there will be no branch showing how you got from local-a to
> local-b, since they're actually two totally different things.  In git,
> the ideal case is that if you look at the history of HEAD, you can see
> the *entire* evolution of the product, and using a different
> merge-base for each branch gets in the way of that.
>
> The best option above, IMHO, is my proposed option 0.  But any of the
> others will work.

Thank you.

j.

Re: Dealing with an upstream cherry-picked branch

[Avery Pennarun]

On Mon, Mar 15, 2010 at 4:07 PM, Jay Soffian <jaysoffian@gmail.com> wrote:
> On Mon, Mar 15, 2010 at 2:46 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
>>> Jay Soffian wrote:
>>> (c) Create a new branch at point Ma and cherry-pick only the local
>>> commits from local-master past point Ma. This essentially gives me the
>>> clean local-master I would've had if I'd been doing (1) all along.
>>
>> This will work, but the short form for exactly the same operation is:
>>
>>        git checkout -b local-b local-master
>>        git rebase --onto upstream-master upstream-a
>
> Are you sure that will skip the merges into local-master from
> upstream-a? I didn't think so and it's not my recollection based on
> using --onto previously.

It should.  The --onto option is rather confusing to get right, and
because git is pretty smart, sometimes the rebase will end up doing
what you want (maybe with some extra conflicts) *despite* using the
wrong (or no) --onto option.

Basically, the 'upstream-a' option above means "remove everything
that's in upstream-a" and then the --onto option means "and then paste
what's left on top of upstream-master".  Since your merge commits only
add things from upstream-a, they too should be left out of the rebase.

Sometimes I find this gets confusing.  You can often sort it out by
using 'rebase -i' (with otherwise the same options as above) and
tweaking it a little.  But the upstream-a commits should definitely
all be gone.

Have fun,

Avery

Re: Dealing with an upstream cherry-picked branch

[Junio C Hamano]

Jay Soffian <jaysoffian@gmail.com> writes:

> I have the following scenario:
>
>   o---o---Ma---o---o    local-master
>  /       /
> |       | .-b'------d'  upstream-a
> |       |/  :       :
> o---o---a---b---c---d   upstream-master
>
> Local-master branched from upstream-master in distant past.
> upstream-master periodically cuts tentative release branch upstream-a.
> When they do this, that branch point (a) is merged into local-master
> (Ma).
>
> Over time, upstream applies fixes to upstream-a, but does so by
> committing the fixes to upstream-master and then cherry-picking them
> to upstream-a.
>
> The question is how to best integrate the fixes on upstream-a into
> local-master, w/o causing a headache when upstream cuts the next
> tentative release branch, at which point upstrea-master will again
> need to be merged into local-master (and that will also have other
> local development). Here are two options I've considered:

Is it an option not to treat "local-master" as anything but a throw-away
integration testing branch?

Presumably upstream-master would be a stable, non-rewinding branch, and
when they branch for their next release (i.e. at point 'a' in the
picture), you should be able to trust up to that commit.  So you would
arrange topics on your side to be based on commits before 'a', and never
let your people fork from or build directly on local-master.

Rebuild local-master every time you would want to run integration testing
with the recent upstream changes, either by starting the throw-away
local-master at upstream-master or upstream-a depending on what
development phase the upstream is in and what the targetted base for your
own release, and merging your topics you intend to ship into it.

                  your topics branch from local-base, which was forked
                  from the last known stable point of the upstream.
                    /       \   \   \
                   /         \   \   \  merge topics for integration testing
                  /           \   \   \
           ...---o local-base  x---x---x local-master (throw-away)
                /             /
    ---o---o---a---b----c----d
                \            upstream-master
                 b'--c'--d'
                         upstream-a

To keep track of the set of topics you know you may want to include in the
upcoming release, you could also merge "well cooked" topics into
local-base so that your people can build on top of other topics and test
them together.

                  your topics branch from (and possibly
                  merge into) local-base
                    / \   \       \   \ 
                   /   \   x---x---x---x local-master (throw-away) 
                  /     \ /   \
 local-base ..---o---o---o   \
                /           /
    ---o---o---a---b---c---d
                \          upstream-master
                 b'--c'--d'  
                         upstream-a

Again, when running integration testing with the recent upstream, you can
rebuild your throw-away local-master at the tip of local-base, and merge
either upstream-master or upstream-a to make sure your changes work with
them.

Individual developers can do the same when testing their own topics in
isolation, together with recent upstream, using their own throw-away
testing branches that merge their topic and recent upstream.

Re: Dealing with an upstream cherry-picked branch

[Jay Soffian]

On Mon, Mar 15, 2010 at 5:30 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Jay Soffian <jaysoffian@gmail.com> writes:
>
>> I have the following scenario:
>>
>>   o---o---Ma---o---o    local-master
>>  /       /
>> |       | .-b'------d'  upstream-a
>> |       |/  :       :
>> o---o---a---b---c---d   upstream-master
>>
>> Local-master branched from upstream-master in distant past.
>> upstream-master periodically cuts tentative release branch upstream-a.
>> When they do this, that branch point (a) is merged into local-master
>> (Ma).
>>
>> Over time, upstream applies fixes to upstream-a, but does so by
>> committing the fixes to upstream-master and then cherry-picking them
>> to upstream-a.
>>
>> The question is how to best integrate the fixes on upstream-a into
>> local-master, w/o causing a headache when upstream cuts the next
>> tentative release branch, at which point upstrea-master will again
>> need to be merged into local-master (and that will also have other
>> local development). Here are two options I've considered:
>
> Is it an option not to treat "local-master" as anything but a throw-away
> integration testing branch?

Our development is currently very fast paced among a smallish group of
developers, and so far the developers really prefer to all work on the
same branch (i.e, not use topic branches). So they are constantly
pushing their changes to local-master. However, I don't think that
precludes an integration branch if I just squint at your picture
below...

> Presumably upstream-master would be a stable, non-rewinding branch, and
> when they branch for their next release (i.e. at point 'a' in the
> picture), you should be able to trust up to that commit.  So you would
> arrange topics on your side to be based on commits before 'a', and never
> let your people fork from or build directly on local-master.
>
> Rebuild local-master every time you would want to run integration testing
> with the recent upstream changes, either by starting the throw-away
> local-master at upstream-master or upstream-a depending on what
> development phase the upstream is in and what the targetted base for your
> own release, and merging your topics you intend to ship into it.
>
>                  your topics branch from local-base, which was forked
>                  from the last known stable point of the upstream.
>                    /       \   \   \
>                   /         \   \   \  merge topics for integration testing
>                  /           \   \   \
>           ...---o local-base  x---x---x local-master (throw-away)
>                /             /
>    ---o---o---a---b----c----d
>                \            upstream-master
>                 b'--c'--d'
>                         upstream-a
>
> To keep track of the set of topics you know you may want to include in the
> upcoming release, you could also merge "well cooked" topics into
> local-base so that your people can build on top of other topics and test
> them together.
>
>                  your topics branch from (and possibly
>                  merge into) local-base
>                    / \   \       \   \
>                   /   \   x---x---x---x local-master (throw-away)
>                  /     \ /   \
>  local-base ..---o---o---o   \
>                /           /
>    ---o---o---a---b---c---d
>                \          upstream-master
>                 b'--c'--d'
>                         upstream-a

By squinting, I can pretend that all the developers pushes to
local-master are mini "well cooked" topics. And I can call my
throw-away branch something like, oh say, "next". So I will have this
picture:

                 developer pushes
                         / \
                    -----   \
                   /   / \   \
local-master ..---o---o---o---o ...    local-master
                 / \       \   \
                /   `---o---o---o---o  next
               /       /           /
       ---o---a---b---c---d---e---f upstream-master
               \
                b'--c'--d' upstream-a


The merges from upstream-master to next will happen at well-defined
times by an integration manager. The merges from local-master to next
will be done by developers themselves to give them exposure to what is
happening upstream-master.

But, it's not clear how that helps me with upstream-a, unless I also
have a "local-a" branch:

local-master ..---o---o---o---o ...    local-master
                 / \       \   \
                /   `---o---o---o---o  local-a
               /       /           /
       ---o---a---b'--c'----------d'   upstream-a


But now I'm asking for developers to:

a) commit changes to local-master
b) merge to next
c) merge to local-a

> Again, when running integration testing with the recent upstream, you can
> rebuild your throw-away local-master at the tip of local-base, and merge
> either upstream-master or upstream-a to make sure your changes work with
> them.
>
> Individual developers can do the same when testing their own topics in
> isolation, together with recent upstream, using their own throw-away
> testing branches that merge their topic and recent upstream.

I think I'm not clear how it helps to make the integration branch
throw-away. Unless you mean something like git pu?

Let's assume I want the developers to have the benefit of seeing
what's in upstream-a and upstream-master. Let's also assume the merges
from upstream-master are typically non-trivial. If "next" in my
picture above is something each developer creates themselves, then
there is no way to share merge resolutions. That would be painful.

j.

Re: Dealing with an upstream cherry-picked branch

[Junio C Hamano]

Jay Soffian <jaysoffian@gmail.com> writes:

> I think I'm not clear how it helps to make the integration branch
> throw-away. Unless you mean something like git pu?

The whole idea of making it throw-away came from that you do not have
control over your upstream and you would want to work with their
upstream-master and "upstream-a of the week".

I think your squinted version essentially is the same as what I outlined,
modulo that your "local-master" corresponds to what I called "local-base".
No matter what we call that branch, as long as you need to be able to
integrate cleanly with both upstream-master and the upstream-a (that is
built by cherry-picking and then thrown away from time to time) and able
to move forward without too much cruft in your history, "local-base" needs
to stay away from merging upstream-a.