sending changesets from the middle of a git tree

sending changesets from the middle of a git tree

[Steve French]

Just to confirm a recent answer to questions on lkml ...

1) There is no way to send a particular changeset from the "middle" of a 
set from one tree to another, without exporting it as a patch or 
rebuilding a new git tree.   I have two changesets that, after testing 
last week, I now consider more important to send upstream than the few 
earlier and later changesets.   If I export those two changesets as 
patches, and send them on. presumably I lose the changset comments etc. 
and then when the upstream tree is merged back, it might look a little 
odd in the changeset history.

2) There is no way to update the comment field of a changeset after it 
goes in (e.g. to add a bugzilla bug number for a bug that was opened 
just after the fix went in).

3) There is no way to do a test commit of an individual changeset 
against a specified tree (to make sure it would still merge cleanly, 
automatically).

Are there easier ways to do any of these?

Re: sending changesets from the middle of a git tree

[Ryan Anderson]

On Sat, Aug 13, 2005 at 10:35:50PM -0500, Steve French wrote:
> Just to confirm a recent answer to questions on lkml ...
> 
> 1) There is no way to send a particular changeset from the "middle" of a 
> set from one tree to another, without exporting it as a patch or 
> rebuilding a new git tree.   I have two changesets that, after testing 
> last week, I now consider more important to send upstream than the few 
> earlier and later changesets.   If I export those two changesets as 
> patches, and send them on. presumably I lose the changset comments etc. 
> and then when the upstream tree is merged back, it might look a little 
> odd in the changeset history.

You can keep most of the metadata you want with "git format-patch".

Extract the changes you need, mail them off.

When you later merge things back together, it should be a trivial merge,
hopefully.

> 2) There is no way to update the comment field of a changeset after it 
> goes in (e.g. to add a bugzilla bug number for a bug that was opened 
> just after the fix went in).

No, a commit is immutable.  You can use "git format-patch" to rebase things if
you need.  I prefer to use "git format-patch --mbox", edit what I need
to, then use git-applymbox to rebase it all against a clean tree.

> 3) There is no way to do a test commit of an individual changeset 
> against a specified tree (to make sure it would still merge cleanly, 
> automatically).

Not sure on this one - in this case, it almost sounds like you want the
feature set of StGit, and/or quilt.  (If "quilt push" succeeds, clearly
it still merges cleanly.)


-- 

Ryan Anderson
  sometimes Pug Majere

Re: sending changesets from the middle of a git tree

[Ryan Anderson]

On Sun, Aug 14, 2005 at 12:02:33AM -0400, Ryan Anderson wrote:
> On Sat, Aug 13, 2005 at 10:35:50PM -0500, Steve French wrote:
>
> > 2) There is no way to update the comment field of a changeset after it 
> > goes in (e.g. to add a bugzilla bug number for a bug that was opened 
> > just after the fix went in).
> 
> No, a commit is immutable.  You can use "git format-patch" to rebase things if
> you need.  I prefer to use "git format-patch --mbox", edit what I need
> to, then use git-applymbox to rebase it all against a clean tree.

Note (and I should have said this at first), doing it this way gives you
the opportunity to combine a few changes if you want.

Use git-format-patch-script to pull out what you want, delete the diff
from the bottom of the file, manually diff what you need using
git-diff-script, and stick that back at the end of the file
git-format-patch-script created.

This seems to be the easiest way to clean up your change history.

-- 

Ryan Anderson
  sometimes Pug Majere

Re: sending changesets from the middle of a git tree

[Linus Torvalds]

On Sat, 13 Aug 2005, Steve French wrote:
> 
> 1) There is no way to send a particular changeset from the "middle" of a 
> set from one tree to another, without exporting it as a patch or 
> rebuilding a new git tree.

Correct.

> If I export those two changesets as patches, and send them on.
> presumably I lose the changset comments etc.

Well, you can export them with "git send-email" and you won't be losing 
any comments.

Alternatively, use "git cherry", which helps re-order the commits in your
tree. They'll be _new_ commits, but they'll have the contents moved over. 
Junio, maybe you want to talk about how you move patches from your "pu" 
branch to the real branches.

> and then when the upstream tree is merged back, it might look a little
> odd in the changeset history.

Well, you'll end up having the same change twice. It happens. Or if you 
just redo your tree as a separate branch, you can reorder things so that 
you don't have them twice at all.

> 2) There is no way to update the comment field of a changeset after it 
> goes in (e.g. to add a bugzilla bug number for a bug that was opened 
> just after the fix went in).

That's correct. Same things apply: you can move a patch over, and create a 
new one with a modified comment, but basically the _old_ commit will be 
immutable.

The good news is that it means that nobody else can change what you said 
or did either. 

> 3) There is no way to do a test commit of an individual changeset 
> against a specified tree (to make sure it would still merge cleanly, 
> automatically).

Oh, sure, that's certainly very possible, and the git cherry stuff even
helps you do it.  Or use "git-apply --check" to just see if a patch
applies and do your own scripts. 

		Linus

Re: sending changesets from the middle of a git tree

[Linus Torvalds]

On Sat, 13 Aug 2005, Linus Torvalds wrote:

> That's correct. Same things apply: you can move a patch over, and create a 
> new one with a modified comment, but basically the _old_ commit will be 
> immutable.

Let me clarify.

You can entirely _drop_ old branches, so commits may be immutable, but
nothing forces you to keep them. Of course, when you drop a commit, you'll 
always end up dropping all the commits that depended on it, and if you 
actually got somebody else to pull that commit you can't drop it from 
_their_ repository, but undoing things is not impossible.

For example, let's say that you've made a mess of things: you've committed
three commits "old->a->b->c", and you notice that "a" was broken, but you
want to save "b" and "c". What you can do is

	# Create a branch "broken" that is the current code
	# for reference
	git branch broken

	# Reset the main branch to three parents back: this 
	# effectively undoes the three top commits
	git reset HEAD^^^
	git checkout -f

	# Check the result visually to make sure you know what's
	# going on
	gitk --all

	# Re-apply the two top ones from "broken"
	#
	# First "parent of broken" (aka b):
	git-diff-tree -p broken^ | git-apply --index
	git commit --reedit=broken^

	# Then "top of broken" (aka c):
	git-diff-tree -p broken | git-apply --index
	git commit --reedit=broken

and you've now re-applied (and possibly edited the comments) the two
commits b/c, and commit "a" is basically gone (it still exists in the
"broken" branch, of course).

Finally, check out the end result again:

	# Look at the new commit history
	gitk --all

to see that everything looks sensible.

And then, you can just remove the broken branch if you decide you really 
don't want it:

	# remove 'broken' branch
	rm .git/refs/heads/broken

	# Prune old objects if you're really really sure
	git prune

And yeah, I'm sure there are other ways of doing this. And as usual, the 
above is totally untested, and I just wrote it down in this email, so if 
I've done something wrong, you'll have to figure it out on your own ;)

			Linus

Re: sending changesets from the middle of a git tree

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

>> If I export those two changesets as patches, and send them on.
>> presumably I lose the changset comments etc.
>
> Well, you can export them with "git send-email" and you won't be losing 
> any comments.

Yes, except the command is "git format-patch".  Not just
comments, but "format-patch --author --date" generates messages
that would help preserving the authorship and original author
date information as well.

> Alternatively, use "git cherry", which helps re-order the commits in your
> tree. They'll be _new_ commits, but they'll have the contents moved over. 

> Junio, maybe you want to talk about how you move patches from your "pu" 
> branch to the real branches.

As I have mentioned elsewhere, I have been trying not to use
JIT, my own Porcelain, to make sure that the core-git barebone
Porcelain is usable.

Unfortunately, this is one area in my workflow that I still
heavily rely on JIT, because it is so handy.  I've kept saying I
do not do Porcelain, but I'll make an exception this time, by
invitation ;-).

Let's say I have three commits in "pu" which are not yet in
"master".  I first use "jit-cherry-snap" to give myself a ready
access to these them:

prompt$ jit-cherry-snap master pu
* 3: Alternate object pool mechanism updates.
* 4: Audit rev-parse users.
* 5: Add cheap local clone '-s' flag to git-clone-script

This command is a thin wrapper around "git cherry".  Instead of
giving you long SHA1 commit object names, JIT gave a short name
to them, #3, #4, and #5 (I can say "git-rev-parse 3", or even
"git-diff-tree 3", for example).  I have to mention that
x"alternate object pool" commit is the oldest (i.e. the one that
immediately follows the "master" head), #4 is its child, and #5
is #4's child.

Suppose I have tested the "pu" branch to my satisfaction, but
I would want to hold the commit #3 back (because I promised
Pasky to hold it for a couple of days).  But I know #4 and #5
are good and would like to push them out.  Here is what I do:

prompt$ git checkout master
prompt$ jit-replay 4 5
*** 4 ***
patching file git-rebase-script
patching file git-reset-script
patching file git-tag-script
*** 5 ***
patching file git-clone-script
prompt$

"jit-replay" is essentially this shell scriptlet:

    #!/bin/sh
    . git-sh-setup-script || die "not a git repository"
    for snap
    do
        snap=$(git-rev-parse --verify "$snap") &&
        git-diff-tree -p "$snap" | git-apply --index &&
        git commit -C "$snap"
    done

That is, to apply the change in an existing commit, and record
it with the commit log message, authorship information and
author timestamp from that commit.  What I just did is to port
commits #4 and #5 on top of the "master" branch.

What happened so far is this:

 * I used to have this commit graph.  O is the original "master"
   head, P is the original "pu" head.

    O --> #3 --> #4 --> #5 == P

 * I advanced the "master" branch, but not along the original
   path that lead to "pu" head.  Instead, I made a fork:


                 * New "master" head
      --> #4'--> #5'
     /
    O --> #3 --> #4 --> #5 == P

As Linus and Ryan already said, the old history is "immutable".
I did not (actually, could not) touch #3...#5 commits; they are
still there.  I created two new commit objects #4' and #5'.  And
#5' is now the new "master" head.

What's left to do is to make sure that I do not lose the change
in #3.  So I rebase the "pu" branch to the new "master" head.
The following sequence does it:

prompt$ git-rev-parse master >.git/refs/heads/pu
prompt$ git checkout pu
prompt$ jit-replay 3
*** 3 ***
patching file cache.h
patching file fsck-cache.c
patching file sha1_file.c

Now, the commit ancestry graph looks like this.  "pu" head is at
P', and the original "pu" head P is still recorded as the
snapshot #5.

                  new "master"
                 *head        *new "pu" head
      --> #4'--> #5'--> #3 == P'
     /
    O --> #3 --> #4 --> #5 == P

As the last sanity check, I make sure that the resulting "pu"
head has exactly the same tree as the original "pu" head;
because all I did in this example was to shuffle the order of 
commits, they should exactly match:

prompt$ git diff HEAD..5
prompt$

And they do match.  I discard the numbered snapshots, because I
do not need them anymore.

prompt$ jit-clean

It is worth pointing out that this workflow means that "pu"
branch is just a staging area and if people start pulling from
and merging with it, things will get messy on the receiving end
(not my end), so the owner of a frequently rebased branch like
this should be very clear about the nature of the branch
upfront.  Saying "this branch will be frequently rebased" is
equivalent of saying "comments and replacement patches are
welcome, but do not merge with it, or you may have hard time
yourself cleaning _your_ history up later".

As I said, I do not do Porcelain, and I would _not_ encourage
people to try JIT out at this point.  Because I have not had
enough time to keep it up-to-date and make it take advantage of
the recent git-core improvements, some of the parts I did not
demonstrate above have still (or have acquired) rough edges that
I myself know not to touch, but others would by accident and
burn themselves.  What I would eventually do is to take good
pieces and ideas out of JIT and repackage them as part of the
core-git barebone Porcelain.

Re: sending changesets from the middle of a git tree

[Petr Baudis]

Dear diary, on Sun, Aug 14, 2005 at 09:57:13AM CEST, I got a letter
where Junio C Hamano <junkio@cox.net> told me that...
> Linus Torvalds <torvalds@osdl.org> writes:
> > Alternatively, use "git cherry", which helps re-order the commits in your
> > tree. They'll be _new_ commits, but they'll have the contents moved over. 
> 
> > Junio, maybe you want to talk about how you move patches from your "pu" 
> > branch to the real branches.
> 
> As I have mentioned elsewhere, I have been trying not to use
> JIT, my own Porcelain, to make sure that the core-git barebone
> Porcelain is usable.
> 
> Unfortunately, this is one area in my workflow that I still
> heavily rely on JIT, because it is so handy.  I've kept saying I
> do not do Porcelain, but I'll make an exception this time, by
> invitation ;-).

Actually, wouldn't this be also precisely for what StGIT is intended to?

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
If you want the holes in your knowledge showing up try teaching
someone.  -- Alan Cox

Re: sending changesets from the middle of a git tree

[Junio C Hamano]

Petr Baudis <pasky@suse.cz> writes:

> Dear diary, on Sun, Aug 14, 2005 at 09:57:13AM CEST, I got a letter
> where Junio C Hamano <junkio@cox.net> told me that...
>> Linus Torvalds <torvalds@osdl.org> writes:
>> 
>> > Junio, maybe you want to talk about how you move patches from your "pu" 
>> > branch to the real branches.
>> 
> Actually, wouldn't this be also precisely for what StGIT is intended to?

Exactly my feeling.  I was sort of waiting for Catalin to speak
up.  With its basing philosophical ancestry on quilt, this is
the kind of task StGIT is designed to do.

I just have done a simpler one, this time using only the core
GIT tools.

I had a handful commits that were ahead of master in pu, and I
wanted to add some documentation bypassing my usual habit of
placing new things in pu first.  At the beginning, the commit
ancestry graph looked like this:

                             *"pu" head
    master --> #1 --> #2 --> #3

So I started from master, made a bunch of edits, and committed:

    $ git checkout master
    $ cd Documentation; ed git.txt git-apply-patch-script.txt ...
    $ cd ..; git add Documentation/*.txt
    $ git commit -s -v

NOTE.  The -v flag to commit is a handy way to make sure that
your additions are not introducing bogusly formatted lines.

After the commit, the ancestry graph would look like this:

                              *"pu" head
    master^ --> #1 --> #2 --> #3
          \
            \---> master

The old master is now master^ (the first parent of the master).
The new master commit holds my documentation updates.

Now I have to deal with "pu" branch.

This is the kind of situation I used to have all the time when
Linus was the maintainer and I was a contributor, when you look
at "master" branch being the "maintainer" branch, and "pu"
branch being the "contributor" branch.  Your work started at the
tip of the "maintainer" branch some time ago, you made a lot of
progress in the meantime, and now the maintainer branch has some
other commits you do not have yet.  And "git rebase" was written
with the explicit purpose of helping to maintain branches like
"pu".  You _could_ merge master to pu and keep going, but if you
eventually want to cherrypick and merge some but not necessarily
all changes back to the master branch, it often makes later
operations for _you_ easier if you rebase (i.e. carry forward
your changes) "pu" rather than merge.  So I ran "git rebase":

    $ git checkout pu
    $ git rebase master pu

What this does is to pick all the commits since the current
branch (note that I now am on "pu" branch) forked from the
master branch, and forward port these changes.

    master^ --> #1 --> #2 --> #3
          \                                  *"pu" head
            \---> master --> #1' --> #2' --> #3'

The diff between master^ and #1 is applied to master and
committed to create #1' commit with the commit information (log,
author and date) taken from commit #1.  On top of that #2' and #3'
commits are made similarly out of #2 and #3 commits.

Old #3 is not recorded in any of the .git/refs/heads/ file
anymore, so after doing this you will have dangling commit if
you ran fsck-cache, which is normal.  After testing "pu", you
can run "git prune" to get rid of those original three commits.

While I am talking about "git rebase", I should talk about how
to do cherrypicking using only the core GIT tools.

Let's go back to the earlier picture, with different labels.

You, as an individual developer, cloned upstream repository and
amde a couple of commits on top of it.

                              *your "master" head
   upstream --> #1 --> #2 --> #3

You would want changes #2 and #3 incorporated in the upstream,
while you feel that #1 may need further improvements.  So you
prepare #2 and #3 for e-mail submission.

    $ git format-patch master^^ master

This creates two files, 0001-XXXX.txt and 0002-XXXX.txt.  Send
them out "To: " your project maintainer and "Cc: " your mailing
list.  You could use contributed script git-send-email-script if
your host has necessary perl modules for this, but your usual
MUA would do as long as it does not corrupt whitespaces in the
patch.

Then you would wait, and you find out that the upstream picked
up your changes, along with other changes.

   where                      *your "master" head
  upstream --> #1 --> #2 --> #3
    used   \ 
   to be     \--> #A --> #2' --> #3' --> #B --> #C
                                                *upstream head

The two commits #2' and #3' in the above picture record the same
changes your e-mail submission for #2 and #3 contained, but
probably with the new sign-off line added by the upsteam
maintainer and definitely with different committer and ancestry
information, they are different objects from #2 and #3 commits.

You fetch from upstream, but not merge.

    $ git fetch upstream

This leaves the updated upstream head in .git/FETCH_HEAD but
does not touch your .git/HEAD nor .git/refs/heads/master.  
You run "git rebase" now.

    $ git rebase FETCH_HEAD master

Earlier, I said that rebase applies all the commits from your
branch on top of the upstream head.  Well, I lied.  "git rebase"
is a bit smarter than that and notices that #2 and #3 need not
be applied, so it only applies #1.  The commit ancestry graph
becomes something like this:

   where                     *your old "master" head
  upstream --> #1 --> #2 --> #3
    used   \                      your new "master" head*
   to be     \--> #A --> #2' --> #3' --> #B --> #C --> #1'
                                                *upstream
                                                head

Again, "git prune" would discard the disused commits #1-#3 and
you continue on starting from the new "master" head, which is
the #1' commit.

-jc

Re: sending changesets from the middle of a git tree

[Catalin Marinas]

Junio C Hamano <junkio@cox.net> wrote:
> Petr Baudis <pasky@suse.cz> writes:
>
>> Dear diary, on Sun, Aug 14, 2005 at 09:57:13AM CEST, I got a letter
>> where Junio C Hamano <junkio@cox.net> told me that...
>>> Linus Torvalds <torvalds@osdl.org> writes:
>>> 
>>> > Junio, maybe you want to talk about how you move patches from your "pu" 
>>> > branch to the real branches.
>>> 
>> Actually, wouldn't this be also precisely for what StGIT is intended to?
>
> Exactly my feeling.  I was sort of waiting for Catalin to speak
> up.  With its basing philosophical ancestry on quilt, this is
> the kind of task StGIT is designed to do.

Have been on holiday and couldn't reply. I will follow up.

-- 
Catalin

Re: sending changesets from the middle of a git tree

[Wolfgang Denk]

Hello,

in message <7vfytc9dzw.fsf@assigned-by-dhcp.cox.net> you wrote:
> 
> This is the kind of situation I used to have all the time when
> Linus was the maintainer and I was a contributor, when you look
> at "master" branch being the "maintainer" branch, and "pu"
> branch being the "contributor" branch.  Your work started at the
...
> your changes) "pu" rather than merge.  So I ran "git rebase":
> 
>     $ git checkout pu
>     $ git rebase master pu

How do you handle conflicts in such a situation?

For example, I get:

	-> git rebase master testing-NAND
	67a002cbe2b2850d76d797e679bc290a76666df6
	patching file common/cmd_nand.c
	Hunk #2 FAILED at 18.
	Hunk #3 succeeded at 335 (offset 1 line).
	1 out of 4 hunks FAILED -- saving rejects to file common/cmd_nand.c.rej
	external diff died, stopping at common/cmd_nand.c.
	Some commits could not be rebased, check by hand:
	67a002cbe2b2850d76d797e679bc290a76666df6

OK, I can edit the file to resolve the conflicts. But what  do  I  do
then to continue?

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
...when fits of creativity run strong, more than  one  programmer  or
writer  has  been  known to abandon the desktop for the more spacious
floor.                                             - Fred Brooks, Jr.

Re: sending changesets from the middle of a git tree

[Junio C Hamano]

Wolfgang Denk <wd@denx.de> writes:

> 	external diff died, stopping at common/cmd_nand.c.
> 	Some commits could not be rebased, check by hand:
> 	67a002cbe2b2850d76d797e679bc290a76666df6
>
> OK, I can edit the file to resolve the conflicts. But what  do  I  do
> then to continue?

Your rebase failed while processing 67a002... and fortunately
(well, if you are really fortunate you would not have seem it)
"some commits could not be" message says that was the only one
left.  At that point, everything up to that commit should be
migrated on top of your head.  Run gitk or git show-branches to
see where the "master" head is.  It should be pointing at the
last successful commit the rebase process has made.  I guess in
your example you had one commit between master and testing-NAND,
in which case the master head does not have moved?

Anyway, you manually resolve conflicts and make an commit from
there, which would migrate the failed 67a002... commit on top of
your master branch.  If "some commits could not be rebased"
message says more than one commit, you need to deal with the
rest of them by hand; sorry there is no automated way currently.

You have to repeat for each such commit:

 (1) Run "git-diff-tree -p <commit-id> | git-apply --index" (or
     what is in git-rebase-script which uses git-apply-patch-script);
 (2) Resolve conflict if there is one;
 (3) Run "git commit -C <commit-id>".

Re: sending changesets from the middle of a git tree

[Catalin Marinas]

There are other ways to do these, explained in this thread. I will
only show the StGIT way, just choose which one suits you better.

Steve French <smfrench@austin.rr.com> wrote:
> 1) There is no way to send a particular changeset from the "middle"
> of a set from one tree to another, without exporting it as a patch
> or rebuilding a new git tree.  I have two changesets that, after
> testing last week, I now consider more important to send upstream
> than the few earlier and later changesets.

With StGIT, you create a new patch ('stg new <name>'), modify and
commit the changes with 'stg refresh'. All the modifications to a
patch are stored as a single GIT commit. If you manage a (contributor)
tree with StGIT, you shouldn't commit changes directly with GIT but
use the StGIT commands instead. You end up with a stack of changesets
on top of the main tree.

You can send the changesets upstream with the 'stg mail' command or
export them with 'stg export'.

> If I export those two changesets as patches, and send them
> on. presumably I lose the changset comments etc. and then when the
> upstream tree is merged back, it might look a little odd in the
> changeset history.

Pulling the latest changes from the main tree will keep your changes
on top, much like git cherry/rebase, but StGIT does a diff3 merge
instead of simply generating and applying patch. This has the
advantage of detecting when a patch (changeset) was not fully merged
or was modifed. If the upstream merge was complete, StGIT shows your
patch as empty (since your patch no longer needs to change the
tree). Otherwise, you can either have some changes in the patch or
even be notified of a conflict (patch modified before being merged).

> 2) There is no way to update the comment field of a changeset after
> it goes in (e.g. to add a bugzilla bug number for a bug that was
> opened just after the fix went in).

'stg refresh --edit' lets you modify the patch text. Since the GIT
commits are immutable, a new commit is generated but the parent of the
new commit is the same as the parent of the old commit (making this
commit unaccessible). Being able to create your own DAG structure with
GIT is what made StGIT possible.

> 3) There is no way to do a test commit of an individual changeset
> against a specified tree (to make sure it would still merge cleanly,
> automatically).

With StGIT you can pop all the patches from the stack and only push
the one you want to test (the push/pop operations also allow patch
reordering). Note that the push operation is done with a three-way
merge and, if successful, the patch might have a sligthly different
form (different offsets for example, or even chunks removed if they
are already in the tree).

If the push fails, it means that it doesn't apply cleanly because it
depends on changes made by other patches in your series. You can undo
the push operation with 'stg push --undo'.

-- 
Catalin