Best way to generate a git tree containing only a subset of commits from another tree?

Best way to generate a git tree containing only a subset of commits from another tree?

[Anton Altaparmakov]

As subject, what is at present the best way to generate a git tree 
containing only a subset of commits from another tree.

So I have /usr/src/my-big-tree and /usr/src/linux-2.6 and now I want to 
add some of the commits in my-big-tree to the tree linux-2.6 so I can push 
out to Linus.

Preferable I would like to do it so that later when Linus has pulled from 
my /usr/src/linux-2.6 tree, I do a "git pull" of Linus' tree from 
/usr/src/my-big-tree and it all works correctly and I don't end up with 
the same commits twice.

Is that possible at all?

If not what can I do to do it cleanly?  Does git help in any way or do I 
literally have to export all my commits from /usr/src/my-big-tree to diff 
style patches and then throw away the tree, clone Linus tree after he has 
pulled my /usr/src/linux-2.6 tree and commit all my generated diff patches 
again?  That would be rather horrible to have to do...

I am happy to be pointed to a FAQ or RTFM if you tell me where to look for 
it...

Thanks a lot in advance!

PS. Please keep me CC:-ed as I am not on the git mailing list any more.

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: Best way to generate a git tree containing only a subset of commits from another tree?

[Radoslaw Szkodzinski]

[-- Attachment #1: Type: text/plain, Size: 995 bytes --]

On Wednesday 22 March 2006 20:28, Anton Altaparmakov wrote yet:
> Preferable I would like to do it so that later when Linus has pulled from
> my /usr/src/linux-2.6 tree, I do a "git pull" of Linus' tree from
> /usr/src/my-big-tree and it all works correctly and I don't end up with
> the same commits twice.
>
> Is that possible at all?

Should work out of the box.

> If not what can I do to do it cleanly?  Does git help in any way or do I
> literally have to export all my commits from /usr/src/my-big-tree to diff
> style patches and then throw away the tree, clone Linus tree after he has
> pulled my /usr/src/linux-2.6 tree and commit all my generated diff patches
> again?  That would be rather horrible to have to do...

It will work flawlessly if Linus merges your patch without any changes.
Else git will merge and maybe conflict if the change was major.

-- 
GPG Key id:  0xD1F10BA2
Fingerprint: 96E2 304A B9C4 949A 10A0  9105 9543 0453 D1F1 0BA2

AstralStorm

[-- Attachment #2: Type: application/pgp-signature, Size: 191 bytes --]

Re: Best way to generate a git tree containing only a subset of commits from another tree?

[Petr Baudis]

Dear diary, on Wed, Mar 22, 2006 at 08:28:52PM CET, I got a letter
where Anton Altaparmakov <aia21@cam.ac.uk> said that...
> Preferable I would like to do it so that later when Linus has pulled from 
> my /usr/src/linux-2.6 tree, I do a "git pull" of Linus' tree from 
> /usr/src/my-big-tree and it all works correctly and I don't end up with 
> the same commits twice.
> 
> Is that possible at all?

Not with Git - you will end up with the same commits twice, once when
you originally committed them and once coming cherry-picked from your
linux-2.6 tree through Linus' tree.

> If not what can I do to do it cleanly?  Does git help in any way or do I 
> literally have to export all my commits from /usr/src/my-big-tree to diff 
> style patches and then throw away the tree, clone Linus tree after he has 
> pulled my /usr/src/linux-2.6 tree and commit all my generated diff patches 
> again?  That would be rather horrible to have to do...

Yes, that's the way to go, but actually it's not horrible at all because
there's a tool to help you - check out StGIT, which will let you
maintain a stack of patches on top of a git tree and do all sorts of
cool stuff with them (including rebasing them to new tree revision, the
most important thing for you).

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Right now I am having amnesia and deja-vu at the same time.  I think
I have forgotten this before.

Re: Best way to generate a git tree containing only a subset of commits from another tree?

[Andreas Ericsson]

Anton Altaparmakov wrote:
> As subject, what is at present the best way to generate a git tree 
> containing only a subset of commits from another tree.
> 

git format-patch -k <start-commit>..<end-commit> --stdout | git am -k

Make sure you're on the right branch first, but see below.

> So I have /usr/src/my-big-tree and /usr/src/linux-2.6 and now I want to 
> add some of the commits in my-big-tree to the tree linux-2.6 so I can push 
> out to Linus.
> 

I sense some nomenclature confusion here. By "tree", do you happen to 
mean "branch", or possibly "repository"? I know bk does things with 
trees that's vaguely git-ish, but a tree in git is, basically, just a 
simplified directory listing (without depth, although it can contain 
other trees ad infinitum iiuc).

If you mean "branch" (or repository, it doesn't matter since by then 
you'll have something like a master branch anyway), as I think you do, 
then let's assume the Vanilla Linux lives in the "linus" branch.

You would then do

    $ git checkout -b for-linus linus

followed by either multiple
    $ git cherry-pick <commit-ish>

or, if the commits are all in series, an iteration of the following

    $ git format-patch --stdout <start-commit>..<end-commit> | git am -k

If you have several topic branches, one for each series of commits, you 
should be able to do an octopus, like so:
    $ git pull . <topic-branches-to-publish>

If you *don't* have several topic branches, or if some commits aren't in 
topic-branches, you could try something like this (untested, although it 
shouldn't break anything except the for-linus branch which you can 
re-create fairly simply)

   $ for b in <topic-branches-for-linus>; do
       git checkout $b
       git rebase for-linus || (git reset --hard; echo $b >> to-merge)
     done
   # now merge what couldn't be rebased
   $ git checkout for-linus
   $ git pull . $(cat to-merge)

In your "please-pull" mail to Linus, ask him to pull the 'for-linus' 
branch. When he's done so, pull his 'master' branch to your 'linus', 
branch and remove the 'for-linus' branch and re-create it from Linus' 
master branch again. If your vanilla tree is up-to-date and he pulls 
from you before pulling from someone else or adding other commits this 
isn't necessary, although you'll have to do
    $ git checkout linus; git pull . for-linus

to get the vanilla branch up to speed with Linus' HEAD.


That turned out a bit longer than I expected, and with me being slightly 
off sobriety at the moment it would be good if someone less so could 
double-check the thinking. Incidentally, this is one of the reasons why 
topic-branches is such a Good Thing (tm).


> Preferable I would like to do it so that later when Linus has pulled from 
> my /usr/src/linux-2.6 tree, I do a "git pull" of Linus' tree from 
> /usr/src/my-big-tree and it all works correctly and I don't end up with 
> the same commits twice.
> 
> Is that possible at all?
> 

I hope so, or I just spent a good 20 minutes not drinking beer for 
nothing. ;)


> If not what can I do to do it cleanly?  Does git help in any way or do I 
> literally have to export all my commits from /usr/src/my-big-tree to diff 
> style patches and then throw away the tree, clone Linus tree after he has 
> pulled my /usr/src/linux-2.6 tree and commit all my generated diff patches 
> again?  That would be rather horrible to have to do...
> 

It's worth re-iterating:
git format-patch --stdout -k <start-commit>..<end-commit> > patch-series
git am -k patch-series

It will save you a lot of work.

> I am happy to be pointed to a FAQ or RTFM if you tell me where to look for 
> it...
> 

Hopefully I just supplied one that can be re-used with some 
text-mangling by someone capable of being legible and making sense at 
the same time.

For more info, check out the man-pages of the commands above (notably 
the cherry-pick man-page and the pull command's "merge with local" feature).

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: Best way to generate a git tree containing only a subset of commits from another tree?

[Junio C Hamano]

Andreas Ericsson <ae@op5.se> writes:

> You would then do
>
>    $ git checkout -b for-linus linus
>
> followed by either multiple
>
>    $ git cherry-pick <commit-ish>
>
> or, if the commits are all in series, an iteration of the following
>
>    $ git format-patch --stdout <start-commit>..<end-commit> | git am -k

With core git tools these two would be the idiom to use.  It
might be more pleasant to use a specialized tool (such as StGIT)
designed to manage the changes meant for upstream.

> If you have several topic branches, one for each series of commits,
> you should be able to do an octopus, like so:
>
>    $ git pull . <topic-branches-to-publish>

Octopus is orthogonal to the issue at hand.  Further, I suspect
that the original repository by Anton is not that cleanly
organized to have such topic branches -- otherwise the question
would not have come up to begin with.

> If you *don't* have several topic branches, or if some commits aren't
> in topic-branches, you could try something like this (untested,
> although it shouldn't break anything except the for-linus branch which
> you can re-create fairly simply)
>
>   $ for b in <topic-branches-for-linus>; do
>       git checkout $b
>       git rebase for-linus || (git reset --hard; echo $b >> to-merge)
>     done

>   # now merge what couldn't be rebased
>   $ git checkout for-linus
>   $ git pull . $(cat to-merge)

Now you lost me here.  When rebase refuses because of
conflicting changes, you are doing "reset --hard" but I suspect
you meant "reset --hard ORIG_HEAD" to recover the original head.
Further, I would have expected you to be rebasing on top of
linus, not for-linus, in case you may already have pulled other
topic branches into it.

Your merging those branches that have conflicting changes on top
of for-linus (that starts out at Linus's tip) is sensible, but
one word of caution is the history contained within the topic
branch should be sane.  What are you going to do with branches
that cleanly rebase on top of for-linus?

> ... If your vanilla tree is up-to-date and he pulls
> from you before pulling from someone else or adding other commits this
> isn't necessary, although you'll have to do
>
>    $ git checkout linus; git pull . for-linus
>
> to get the vanilla branch up to speed with Linus' HEAD.

I am not sure I follow you here.

If Linus hasn't pulled from you, you can either just keep asking
(you do not have to update for-linus), or rebuild it based on
more recent Linus's tip.

	$ git fetch linus ;# to update to Linus's tip
	$ git checkout for-linus
        $ git reset --hard linus

If Linus has pulled from you, there is nothing more than the
above for you to do.  If you want to rebuild for-linus branch,
(maybe because you fixed things in some of your topic branches),
after the above, you could:

        $ git pull . this-topic
        $ git pull . that-topic
        ...

This is nicer to Linus _if_ your topics overlap with recent
changes to the Linus's tree.  Otherwise you do not necessarily
have to rebuild for-linus branch.

Re: Best way to generate a git tree containing only a subset of commits from another tree?

[Andreas Ericsson]

Junio C Hamano wrote:
> Andreas Ericsson <ae@op5.se> writes:
> 
  >>If you have several topic branches, one for each series of commits,
>>you should be able to do an octopus, like so:
>>
>>   $ git pull . <topic-branches-to-publish>
> 
> 
> Octopus is orthogonal to the issue at hand.  Further, I suspect
> that the original repository by Anton is not that cleanly
> organized to have such topic branches -- otherwise the question
> would not have come up to begin with.
> 

<sidenote>
I've never understood what orthogonal means in this sense. "at a right 
angle" as in flagging for attention or the exactly counter-productive to 
what one should use?
</sidenot>

> 
>>If you *don't* have several topic branches, or if some commits aren't
>>in topic-branches, you could try something like this (untested,
>>although it shouldn't break anything except the for-linus branch which
>>you can re-create fairly simply)
>>
>>  $ for b in <topic-branches-for-linus>; do
>>      git checkout $b
>>      git rebase for-linus || (git reset --hard; echo $b >> to-merge)
>>    done
> 
> 
>>  # now merge what couldn't be rebased
>>  $ git checkout for-linus
>>  $ git pull . $(cat to-merge)
> 
> 
> Now you lost me here.  When rebase refuses because of
> conflicting changes, you are doing "reset --hard" but I suspect
> you meant "reset --hard ORIG_HEAD" to recover the original head.


I actually meant to reset the for-linus branch, although it would have 
to be reset to the state it was before trying the rebase, which means 
creating and deleting a tag or some other marker.

I really should install an alco-lock on my MUA.


> Further, I would have expected you to be rebasing on top of
> linus, not for-linus, in case you may already have pulled other
> topic branches into it.
> 

Perhaps. I said 'for-linus' to make sure there was an easy way to 
recover to state 1 in case of errors. I also rewrote the part above 
twice to account for topic branches, so it doesn't make much sense 
without the background thinking.


> Your merging those branches that have conflicting changes on top
> of for-linus (that starts out at Linus's tip) is sensible, but
> one word of caution is the history contained within the topic
> branch should be sane.  What are you going to do with branches
> that cleanly rebase on top of for-linus?
> 

Nothing. 'for-linus' should be updated each time a rebase completes 
success-fully, so all the cleanly rebased branches should be in a linear 
commit-history on top of each other. Granted, most projects won't have 
many topic-branches (or other commit-chains) that rebase on top of each 
other like that, but...

> 
>>... If your vanilla tree is up-to-date and he pulls
>>from you before pulling from someone else or adding other commits this
>>isn't necessary, although you'll have to do
>>
>>   $ git checkout linus; git pull . for-linus
>>
>>to get the vanilla branch up to speed with Linus' HEAD.
> 
> 
> I am not sure I follow you here.
> 
> If Linus hasn't pulled from you, you can either just keep asking
> (you do not have to update for-linus), or rebuild it based on
> more recent Linus's tip.
> 

What I meant was that the thing he has in "for-linus" will match what 
Linus has in "master" verbatim if Linus doesn't have commits on top of 
his "master" that aren't in Anton's "for-linus" (originating from 
"linus"). That sentence didn't make sense to me right now.

> 	$ git fetch linus ;# to update to Linus's tip
> 	$ git checkout for-linus
>         $ git reset --hard linus
> 
> If Linus has pulled from you, there is nothing more than the
> above for you to do.

The above command would reset the "for-linus" branch to the state it had 
before he applied all his changes. I meant that if he wants to track 
Linus' exact HEAD in some branch he could do that by tracking his own if 
the changes since merge-base are identical. I was clearly complicating 
things by mentioning such a highly conditional exception.


>  If you want to rebuild for-linus branch,
> (maybe because you fixed things in some of your topic branches),
> after the above, you could:
> 
>         $ git pull . this-topic
>         $ git pull . that-topic
>         ...
> 
> This is nicer to Linus _if_ your topics overlap with recent
> changes to the Linus's tree.  Otherwise you do not necessarily
> have to rebuild for-linus branch.
> 

But it's very nasty in case Linus has already pulled the changes, which 
was what I assumed he would have done.

I was most likely a bit diffuse. Everything else seems to be at the 
moment, and I like to blend in. ;)

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: Best way to generate a git tree containing only a subset of commits from another tree?

[Linus Torvalds]

On Thu, 23 Mar 2006, Andreas Ericsson wrote:
> 
> <sidenote>
> I've never understood what orthogonal means in this sense. "at a right angle"
> as in flagging for attention or the exactly counter-productive to what one
> should use?
> </sidenot>

No. Orthogonal in math may be literally "straight angle", but in 
non-geometric speak it means "independent" or "statistically unrelated".

See 

	http://wordnet.princeton.edu/perl/webwn?s=orthogonal

and the two first definitions in particular.

Ie two issues (or, in this case, "branches") are orthogonal if they have 
nothing in common - they fix two totally independent things.

This is, btw, totally consistent with the geometric meaning of the word. 
Two vectors are orthogonal if they have no common component: the dot 
product is zero (ie the projection of one vector onto another is the null 
vector).

So if you see two lines of development as being "vectors" from a common 
source, when they have nothing in common, they are orthogonal.

Of course, the development space is neither three-dimensional nor 
euclidian, so it's a strange kind of vector, but still ;)

		Linus