interactive rebase not rebasing

interactive rebase not rebasing

[Stephen Haberman]

[-- Attachment #1: Type: text/plain, Size: 2715 bytes --]

Hello,

Per the emails from me last week, I'm working in an environment with
shared topic branches and am trying to find a bullet-proof way for devs
to rebase their local commits after the topic branch has moved.

The easy approach would be to just let `git pull` create merge commits,
and I would have been done with this long ago, but I'm striving to get
rebasing to "just work" and avoid the ugliness of same-branch merge
commits. Cross-branch merge commits are cool, but not same-branch.

So, here's a crazy scenario we've ran into--a new release has hit
stable, with two devs working on the same topic branch, and both of them
merge. One wins, and the other has to rebase. Previously, this was
replaying commits, but with great tips from the list last week, `rebase
-i -p` is handling most scenarios.

However, not this one:

# A --C------            <-- origin/stable
#  \  |      \
#   B -- D -- E -- F     <-- origin/topic2
#    \|
#     g -- h             <-- topic2

topic2 is a dev that has locally merged stable in g, moved on in h, is
ready to push, but another dev already merged stable in E, and also
committed F.

If we do a `git pull --no-rebase`, the result is:

# A --C------            <-- origin/stable
#  \  |      \
#   B -- D -- E -- F     <-- origin/topic2
#    \|             \
#     g -- h ------- i   <-- topic2

But i is a same-branch merge, and we'd rather rebase to something like:

# A --C------            <-- origin/stable
#  \         \
#   B -- D -- E -- F     <-- origin/topic2
#                   \
#                    h'  <-- topic2

(...maybe g' in there if g resolved stable conflicts differently
E did them. I'm not sure, I haven't gotten there yet.)

However, currently, `git rebase -i -p origin/topic2` results in:

# A --C------            <-- origin/stable
#  \  |      \
#   B -- D -- E -- F     <-- origin/topic2
#    \|
#     g -- h             <-- topic2

Nothing has changed. g & h haven't moved...I can keep executing this
operation and the commits never make it on top of origin/topic2's F. 

Frustratingly, if I run non-interactive rebase, it works perfectly. But
I've got other cases (see earlier posts) that do need the interactive
rebase. Personally, I could probably make do with trying each and
seeing what happened, but I'm really trying to find a bullet proof
command/alias/script for devs to run and have it "just work".

I've attached a test that sets up the DAG as above and currently passes
by asserting the not-desired/unchanged-DAG result. The assert for the
desired result is commented out at the end.

Am I correct in asserting this is some sort of bug in the interactive
rebase such that g & h are not ending up on top of F?

Thanks,
Stephen



[-- Attachment #2: t3404b.sh --]
[-- Type: text/x-sh, Size: 2680 bytes --]

#!/bin/sh

test_description='rebase interactive does not rebase'

. ./test-lib.sh

test_expect_success 'setup' '
	echo "setup" >a &&
	git add a &&
	git commit -m "setup" &&
	git clone ./. server &&
	rm -fr server/.git/hooks &&
	git remote add origin ./server &&
	git config --add branch.master.remote origin &&
	git config --add branch.master.merge refs/heads/master &&
	git fetch &&

	git checkout -b stable master &&
	echo "setup.stable" >a &&
	git commit -a -m "stable" &&
	git push origin stable
'
#
# A --C------            <-- origin/stable
#  \  |      \
#   B -- D -- E -- F     <-- origin/topic2
#    \|             \
#     g -- h ------- i   <-- topic2
#
# Trying to push F..i
#
# merge-base(F, h) has two options: B and C
#
test_expect_success 'merging in stable with tricky double baserev does not fool the script' '
	# B: start our topic2 branch, and share it
	git checkout -b topic2 origin/stable &&
	git config --add branch.topic2.merge refs/heads/topic2 &&
	echo "commit B" >a.topic2 &&
	git add a.topic2 &&
	git commit -m "commit B created topic2" &&
	git push origin topic2 &&

	# C: now, separately, move ahead stable, and share it
	git checkout stable
	echo "commit C" >a &&
	git commit -a -m "commit C moved stable" &&
	git push origin stable &&

	# D: have another client commit (in this case, it is the server, but close enough) moves topic2
	cd server &&
	git checkout topic2 &&
	echo "commit D continuing topic2" >a.client2 &&
	git add a.client2 &&
	git commit -m "commit D by client2" &&

	# E: the same other client merges the moved stable
	git merge stable &&

	# F: the same other client moves topic2 again
	echo "commit F" >a.client2 &&
	git commit -a -m "commit F by client2" &&
	F_hash=$(git rev-parse HEAD) &&
	cd .. &&

	# g: now locally merge in the moved stable (even though our topic2 is out of date)
	git checkout topic2 &&
	git merge stable &&
	g_hash=$(git rev-parse HEAD) &&

	# h: advance local topic2
	echo "commit H" >a.topic2 &&
	git commit -a -m "commit H continues local fork" &&
	h_hash=$(git rev-parse HEAD) &&

	# i: make a new merge commit
	git pull --no-rebase &&
	i_hash=$(git rev-parse HEAD) &&

	# Watch merge rejected as something that should get rebased
	# ! git push origin topic2
	test "$i_hash $h_hash $F_hash" = "$(git rev-list --parents --no-walk HEAD)"

	# Now fix it the merge by rebasing it
	git reset --hard ORIG_HEAD &&
	GIT_EDITOR=: git rebase -i -p origin/topic2 &&
	h2_hash=$(git rev-parse HEAD) &&

	# Should be:
	# test "$h2_hash $F_hash" = "$(git rev-list --parents --no-walk HEAD)"
	# But is just:
	test "$h_hash $g_hash" = "$(git rev-list --parents --no-walk HEAD)"
	# Where did $F_hash go?
'

test_done

Re: interactive rebase not rebasing

[Andreas Ericsson]

Stephen Haberman wrote:
> Hello,
> 
> Per the emails from me last week, I'm working in an environment with
> shared topic branches and am trying to find a bullet-proof way for devs
> to rebase their local commits after the topic branch has moved.
> 
> The easy approach would be to just let `git pull` create merge commits,
> and I would have been done with this long ago, but I'm striving to get
> rebasing to "just work" and avoid the ugliness of same-branch merge
> commits. Cross-branch merge commits are cool, but not same-branch.
> 
> So, here's a crazy scenario we've ran into--a new release has hit
> stable, with two devs working on the same topic branch, and both of them
> merge. One wins, and the other has to rebase. Previously, this was
> replaying commits, but with great tips from the list last week, `rebase
> -i -p` is handling most scenarios.
> 
> However, not this one:
> 
> # A --C------            <-- origin/stable
> #  \  |      \
> #   B -- D -- E -- F     <-- origin/topic2
> #    \|
> #     g -- h             <-- topic2
> 
> topic2 is a dev that has locally merged stable in g, moved on in h, is
> ready to push, but another dev already merged stable in E, and also
> committed F.
> 
> If we do a `git pull --no-rebase`, the result is:
> 
> # A --C------            <-- origin/stable
> #  \  |      \
> #   B -- D -- E -- F     <-- origin/topic2
> #    \|             \
> #     g -- h ------- i   <-- topic2
> 
> But i is a same-branch merge, and we'd rather rebase to something like:
> 
> # A --C------            <-- origin/stable
> #  \         \
> #   B -- D -- E -- F     <-- origin/topic2
> #                   \
> #                    h'  <-- topic2
> 
> (...maybe g' in there if g resolved stable conflicts differently
> E did them. I'm not sure, I haven't gotten there yet.)
> 
> However, currently, `git rebase -i -p origin/topic2` results in:
> 
> # A --C------            <-- origin/stable
> #  \  |      \
> #   B -- D -- E -- F     <-- origin/topic2
> #    \|
> #     g -- h             <-- topic2
> 
> Nothing has changed. g & h haven't moved...I can keep executing this
> operation and the commits never make it on top of origin/topic2's F. 
> 
> Frustratingly, if I run non-interactive rebase, it works perfectly.

I can imagine. Since you don't want to preserve the merges in this
case, you shouldn't be using the -p flag.

In fact, for this particular scenario (assuming "h" is really the only
commit on topic2), you probably want to just cherry-pick that commit
into origin/topic2:

   git checkout topic2
   git reset --hard origin/topic2
   git cherry-pick ORIG_HEAD

would work nicely for that. If topic2 and topic1 resolved merge
conflicts differently, cherry-picking topic2 will give conflicts again,
so they'll have to be resolved. rebase (without -p) will have the same
effect, ofcourse.

I don't think you can have a single command that does all the things
you want, because the possible differences in input makes it very nearly
impossible to always do "the right thing". For this case, you don't
want to preserve merges, so "git rebase" without -p does the right
thing. For that other case earlier this week, you wanted to preserve
merges, so "git rebase -p" does the right thing there. It's complex
to try to dwim things like that, and complexity will cause very weird
and surprising errors when it *does* fail.

Instead, I think you should give your developers some training in the
more advanced parts of git's integration tools and also look over
your workflow. Frequently merging 'master' into a topic is very rarely
the right thing to do. It's usually better to rebase the topic onto
master or cherry-pick relevant bugfixes from 'master' into the topic.
If two devs need to work on one topic, then that topic probably needs
breaking up into a sub-topic. However, sub-topics should never merge
'master' into them themselves; only 'topic'. Otherwise you'll get
more merges than work done.


> But
> I've got other cases (see earlier posts) that do need the interactive
> rebase. Personally, I could probably make do with trying each and
> seeing what happened, but I'm really trying to find a bullet proof
> command/alias/script for devs to run and have it "just work".
> 

Or you could sit down and think about what you want to happen and then
run the appropriate command. ;-)

> I've attached a test that sets up the DAG as above and currently passes
> by asserting the not-desired/unchanged-DAG result. The assert for the
> desired result is commented out at the end.
> 

Thanks for this. If all bug-reports had this, bugfixing would be a lot
easier. However, for this particular case, I'm not so sure it's a bug.

> Am I correct in asserting this is some sort of bug in the interactive
> rebase such that g & h are not ending up on top of F?
> 

Assuming you're passing a correct input file to rebase -i; yes. At the
very least, "h" should be moved to the tip of origin/topic2.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: interactive rebase not rebasing

[Stephen Haberman]

> > # A --C------            <-- origin/stable
> > #  \  |      \
> > #   B -- D -- E -- F     <-- origin/topic2
> > #    \|
> > #     g -- h             <-- topic2
> > 
> > Nothing has changed. g & h haven't moved...I can keep executing this
> > operation and the commits never make it on top of origin/topic2's F. 
> > 
> > Frustratingly, if I run non-interactive rebase, it works perfectly.
> 
> I can imagine. Since you don't want to preserve the merges in this
> case, you shouldn't be using the -p flag.

No, I do want to preserve most merges. This "most" qualification is
because the merge "g", if rebased, would have been a no-op, so `rebase
-i -p` correctly kept it out of the TODO file.

Which is cool, except that later on, when rewriting the other TODO
commits, some of which were children of "g", it did not remember that
"g" had gone away, so did nothing to take "g" out of the rewritten
children's parent list.

> In fact, for this particular scenario (assuming "h" is really the only
> commit on topic2), you probably want to just cherry-pick that commit
> into origin/topic2:
> 
>    git checkout topic2
>    git reset --hard origin/topic2
>    git cherry-pick ORIG_HEAD

Agreed. This makes a lot of sense for me, who has been hacking around in
git-rebase--interactive fixing things, but I'd really like the other
people on my team to just have to run `git rebase -i -p`.

> I don't think you can have a single command that does all the things
> you want, because the possible differences in input makes it very
> nearly impossible to always do "the right thing".

Ah, you are too pessimistic. :-)

> Assuming you're passing a correct input file to rebase -i; yes. At the
> very least, "h" should be moved to the tip of origin/topic2.

Cool, agreed. I've got a patch that gets `rebase -i -p` to do this. I'll
send it to the list soon.

- Stephen

Re: interactive rebase not rebasing

[Andreas Ericsson]

Stephen Haberman wrote:
>>> # A --C------            <-- origin/stable
>>> #  \  |      \
>>> #   B -- D -- E -- F     <-- origin/topic2
>>> #    \|
>>> #     g -- h             <-- topic2
>>>
>>> Nothing has changed. g & h haven't moved...I can keep executing this
>>> operation and the commits never make it on top of origin/topic2's F. 
>>>
>>> Frustratingly, if I run non-interactive rebase, it works perfectly.
>> I can imagine. Since you don't want to preserve the merges in this
>> case, you shouldn't be using the -p flag.
> 
> No, I do want to preserve most merges. This "most" qualification is
> because the merge "g", if rebased, would have been a no-op, so `rebase
> -i -p` correctly kept it out of the TODO file.
> 
> Which is cool, except that later on, when rewriting the other TODO
> commits, some of which were children of "g", it did not remember that
> "g" had gone away, so did nothing to take "g" out of the rewritten
> children's parent list.
> 
>> In fact, for this particular scenario (assuming "h" is really the only
>> commit on topic2), you probably want to just cherry-pick that commit
>> into origin/topic2:
>>
>>    git checkout topic2
>>    git reset --hard origin/topic2
>>    git cherry-pick ORIG_HEAD
> 
> Agreed. This makes a lot of sense for me, who has been hacking around in
> git-rebase--interactive fixing things, but I'd really like the other
> people on my team to just have to run `git rebase -i -p`.
> 
>> I don't think you can have a single command that does all the things
>> you want, because the possible differences in input makes it very
>> nearly impossible to always do "the right thing".
> 
> Ah, you are too pessimistic. :-)
> 

Perhaps, although I think you're being overly optimistic if you think
rebase can do all of this intelligently while still retaining clear
semantics. I'd start with writing a separate tool for it, probably
based on git sequencer. When that works out ok, get it to do what
rebase does today and then incorporate the new tool as an option to
rebase and get ready to answer complex questions for the cases where
it doesn't do what the user wanted it to do.

Git is stupid very, very much by design. Linus constantly says that
he prefers tools that he can figure out why they did something stupid
over tools that try really hard to get it right, and I agree with him
100%, as do most of the core contributors (insofar as I've understood
it). What you're suggesting is that a git command should try hard to
dwim a lot of complexity about choosing tools away, and that goes
right against the KISS principle which serves git so well. I'm not
saying you shouldn't do it. Merely that you should think hard about
it and then make sure it doesn't break anything people are already
doing today with the current toolset.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: interactive rebase not rebasing

[Stephen Haberman]

> >> I don't think you can have a single command that does all the
> >> things you want, because the possible differences in input makes it
> >> very nearly impossible to always do "the right thing".
> > 
> > Ah, you are too pessimistic. :-)
> > 
> 
> Perhaps, although I think you're being overly optimistic if you think
> rebase can do all of this intelligently while still retaining clear
> semantics. I'd start with writing a separate tool for it, probably
> based on git sequencer.

I would agree with this, except that --preserve-merges is already in
the codebase and does what we want. I'm not fundamentally changing how
it works (e.g. how it decides what commits to keep/rewrite/etc.), I just
found and patched a bug in it.

> When that works out ok, get it to do what rebase does today and then
> incorporate the new tool as an option to rebase and get ready to
> answer complex questions for the cases where it doesn't do what the
> user wanted it to do.

Yeah, I'm sorry, I did not mean my "pessimistic" comment that
seriously. I understand `git rebase` can never do what everyone wants
in all scenarios.

But given /this/ scenario (hehe), with the implementation's existing
explicit usage of "--left-right --cherry-pick" to drop no-op commits,
but then it's forgetting of this information later, leading to `git
rebase` not performing a rebase at all, I think it is an obvious bug,
and one that can be fixed without changing any of `git rebase`s
existing semantics.

> Merely that you should think hard about it and then make sure it
> doesn't break anything people are already doing today with the current
> toolset.

I've attempted to do that. Now that I sent in the patch, if you could
review it, I would appreciate your feedback. I do need to rework the
test case because I realized including the origin/pull aspects (which
is what caused us to find it) is just noise and the bug can be
reproduced with just local branches.

Thanks,
Stephen

Re: interactive rebase not rebasing

[Andreas Ericsson]

Stephen Haberman wrote:
> 
> But given /this/ scenario (hehe), with the implementation's existing
> explicit usage of "--left-right --cherry-pick" to drop no-op commits,
> but then it's forgetting of this information later, leading to `git
> rebase` not performing a rebase at all, I think it is an obvious bug,
> and one that can be fixed without changing any of `git rebase`s
> existing semantics.
> 

Agreed.

>> Merely that you should think hard about it and then make sure it
>> doesn't break anything people are already doing today with the current
>> toolset.
> 
> I've attempted to do that. Now that I sent in the patch, if you could
> review it, I would appreciate your feedback.

I'm heading home from work now. I'll look it over tonight or tomorrow
morning. Thanks for the confidence :-)

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: interactive rebase not rebasing

[Stephen Haberman]

> > I've attempted to do that. Now that I sent in the patch, if you could
> > review it, I would appreciate your feedback.
> 
> I'm heading home from work now. I'll look it over tonight or tomorrow
> morning.

Cool, thanks.

Question: how taboo is it to just add another test file?

I'm attempting to integrate my test into t3404, which is the existing
interactive rebase test. The two test_expect_success's I added worked
when I ran them at the start of the test and then reset --hard the
branches back for the other tests, but if I paste my tests where they
should probably be, in the middle after the other -p tests, they break
because the 10 or tests before this have screwed with the DAG already.

I can suffer through getting it to work, but a t3409 would be much
easier, and probably easier to read as well as a I could setup my own
DAG instead of hacking onto 3404s.

> Thanks for the confidence :-)

No problem. :-)

- Stephen

Re: interactive rebase not rebasing

[Shawn O. Pearce]

Stephen Haberman <stephen@exigencecorp.com> wrote:
> 
> > > I've attempted to do that. Now that I sent in the patch, if you could
> > > review it, I would appreciate your feedback.
> > 
> > I'm heading home from work now. I'll look it over tonight or tomorrow
> > morning.
> 
> Cool, thanks.
> 
> Question: how taboo is it to just add another test file?
> 
> I can suffer through getting it to work, but a t3409 would be much
> easier, and probably easier to read as well as a I could setup my own
> DAG instead of hacking onto 3404s.

Usually folks prefer to add stuff to an existing test file, but
if the DAG is already a mess and you need a different DAG I find
it easier to just add new test file.  Thus far Junio hasn't pushed
back when I've done that.  Maybe I'm just lucky.  :-)

-- 
Shawn.

Re: interactive rebase not rebasing

[Andreas Ericsson]

Stephen Haberman wrote:
>>> I've attempted to do that. Now that I sent in the patch, if you could
>>> review it, I would appreciate your feedback.
>> I'm heading home from work now. I'll look it over tonight or tomorrow
>> morning.
> 
> Cool, thanks.
> 
> Question: how taboo is it to just add another test file?
> 

Not so taboo. Especially not if there are compelling technical reasons
not to add stuff to an existing one.

> I'm attempting to integrate my test into t3404, which is the existing
> interactive rebase test. The two test_expect_success's I added worked
> when I ran them at the start of the test and then reset --hard the
> branches back for the other tests, but if I paste my tests where they
> should probably be, in the middle after the other -p tests, they break
> because the 10 or tests before this have screwed with the DAG already.
> 

Where you add the tests doesn't matter much. Many tests are grouped by
feature simply because they were added along with the feature they're
testing. There's no other value of grouping tests together.

> I can suffer through getting it to work, but a t3409 would be much
> easier, and probably easier to read as well as a I could setup my own
> DAG instead of hacking onto 3404s.
> 

t3409 is already in spearce.git's next branch. You should be able to
add stuff to that, or add t3410 if that doesn't work so good.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231