How to pull only a few files from one branch to another?

How to pull only a few files from one branch to another?

[Bill Lear]

I have a long-running topic branch.  I have fixed a few nits on
the master branch that I would like on the topic branch.  How do I
pull in only a few files from the head of the master branch?

I tried all sorts of incantations (I am running 1.5.0-rc2), including:

git pull . origin foo.cc
git pull origin foo.cc
git pull foo.cc master
git pull . master foo.cc
git pull master foo.cc


Bill

Re: How to pull only a few files from one branch to another?

[Linus Torvalds]

On Tue, 23 Jan 2007, Bill Lear wrote:
>
> I have a long-running topic branch.  I have fixed a few nits on
> the master branch that I would like on the topic branch.  How do I
> pull in only a few files from the head of the master branch?
> 
> I tried all sorts of incantations (I am running 1.5.0-rc2), including:
> 
> git pull . origin foo.cc
> git pull origin foo.cc
> git pull foo.cc master
> git pull . master foo.cc
> git pull master foo.cc

Git *very* fundamentally tracks project state, not file state. Which means 
that you very much can NOT try to "merge a file". It is a senseless 
operation in git, and in fact, any SCM that allows it pretty much is 
doomed to be a total piece of sh*t (*).

So if you want to get just one file, you cannot see it as a "project-level 
merge", and you literally have to handle that file as an individual file 
and merge it as such, and then you can commit the result as a regular 
commit (but it will _not_ show up as real "git merge").

So in git, the rule is always that you track the whole state of the 
project. It's a project tracker, not a "file tracker".

That said, if you just want to get the fixes from one branch, and merge 
it, git does allow you to do it, although there isn't any "porcelain". 
You'd basically have to do it by hand, a few different ways

 (1) If you just want to commit the state of a file AS IT IS in another 
     branch, you can just do

	git checkout other-branch filename-goes-here
	git commit

     which will just do it for you.

 (2) However, more commonly, what you actually *want* to do is to merge 
     the changes in the other branch into the current branch (which has 
     its own set of changes), and then things get more interesting and 
     more complex.

     If you have just one simple merge-base, you'd do something like

	git merge-one-file \
		$(git merge-base HEAD other):filename \
		HEAD:filename \
		other:filename \
		filename

     (where those last three things are just the file mode). And then just 
     commit the end result.

     Which is just total git magic. If you understand the above command
     line, I have nothing more to teach you.

Maybe somebody could actually test the above one-liner magic thing, and 
explain how it works. And maybe we could write porcelain to do this ;)

NOTE NOTE NOTE! I don't actually think "git merge-one-file" really works 
correctly as-is for your usage case. It _kind_of_ gets the result you 
want, but it's not really written to be used like that, and you'll get an 
error from the

	git-checkout-index -f --stage=2 -- "$4" &&
		cat "$src1" >"$4"

stage because stage 2 didn't exist ("git-merge-one-file" is really only 
supposed to be called by a real merge).

But it's _almost_ usable that way. Maybe Dscho or Junio wants to make it 
work the extra final mile.

Anyway, the above (2) kind of works, but if you want to avoid the warning 
and want to avoid the magic, you would really have to do something like

	git cat-file blob $(git-merge-base HEAD other):filename > .base
	git cat-file blob other:filename > .other
	git merge-file filename .base .other

which will mostly do what you wanted (without complaints), and is what the 
git-merge-one-file magic actually boils down to.

Not exactly pretty. If this is something people want to do often, we'd 
need to generate some porcelain for it.

But if you understand the above three lines, you're actually doing really 
really well already. And as a "yes, git can do it" kind of demonstration, 
it may even be useful.

FINAL NOTE! After you do file-level merging like this, don't expect future 
merges to that file to be easy if you continue to make changes! This is 
_why_ git doesn't do file-level merges natively - namely it really doesn't 
work well together with future merges.

To keep merges working after more than just one merge, you really need to 
either do _all_ merges on a file-per-file level (which is insane and 
doesn't scale and has serious problems) or you need to do merges at a 
"project level" (which is how git fundamentally works).

		Linus

(*) And I'm not saying that just because git doesn't do it. It's much 
more fundamental than that. Once you start doing per-file branching and 
merging, you've basically screwed yourself, and you'll never be able to 
work on the project as a "whole project" any more - you no longer have a 
well-defined history that actually is the history of the whole project.

Re: How to pull only a few files from one branch to another?

[Linus Torvalds]

On Tue, 23 Jan 2007, Linus Torvalds wrote:
> 
>      If you have just one simple merge-base, you'd do something like
> 
> 	git merge-one-file \
> 		$(git merge-base HEAD other):filename \
> 		HEAD:filename \
> 		other:filename \
> 		filename
> 
>      (where those last three things are just the file mode)

That comment made no sense. I removed the file modes entirely, and forgot 
to remove the comment. It turns out that you can just drop the file modes 
entirely from the invocation of least the current "git-merge-one-file", so 
I did that, to keep it cleaner and simpler.

Anyway, the way "git merge-one-file" works, you pass it:

 - the SHA1 of the "base object"

	$(git merge-base HEAD other):filename

 - the SHA1 of the "current branch version of the file"

	HEAD:filename

 - the SHA1 of the "other branch version of the file"

	other:filename

 - the filename as you want the end result to be in the index

	filename

 - normally, you'd pass the file modes for the three cases too

	0644 0644 0644

   but those are the ones I skipped because the shell script actually only 
   uses them to see if they are different (so dropping them means that it 
   all works fine - the empty arguments are all the same).

but additionally, since "git merge-one-file" expects to be in the middle 
of a real merge (and not just some faked-out single-file merging thing) it 
also expects the index to be populated in stage2 with the current branch 
version of the file - which it won't be outside of a merge. Which is why 
it ends up complaining.

It would be kind of cool to have the whole "merge-recursive" logic (which 
can handle multiple ancestors etc - unlike the above) able to do this, but 
that's a whole 'nother kettle of fish.

			Linus

Re: How to pull only a few files from one branch to another?

[Johannes Schindelin]

Hi,

On Tue, 23 Jan 2007, Linus Torvalds wrote:

> It would be kind of cool to have the whole "merge-recursive" logic 
> (which can handle multiple ancestors etc - unlike the above) able to do 
> this, but that's a whole 'nother kettle of fish.

You'd lose all the renaming logic when doing it strictly on a file level. 
So I suggest opening a sidebranch, doing the merge, and just pick the new 
file as you proposed with "git checkout -f sidebranch file".

Ciao,
Dscho

Re: How to pull only a few files from one branch to another?

[Johannes Schindelin]

Hi,

On Tue, 23 Jan 2007, Linus Torvalds wrote:

> NOTE NOTE NOTE! I don't actually think "git merge-one-file" really works 
> correctly as-is for your usage case. It _kind_of_ gets the result you 
> want, but it's not really written to be used like that, and you'll get an 
> error from the
> 
> 	git-checkout-index -f --stage=2 -- "$4" &&
> 		cat "$src1" >"$4"
> 
> stage because stage 2 didn't exist ("git-merge-one-file" is really only 
> supposed to be called by a real merge).
> 
> But it's _almost_ usable that way. Maybe Dscho or Junio wants to make it 
> work the extra final mile.

I don't really see the point, but yes, it would be easy enough (at least 
in the builtin version I sent out last night).

Note that the checkout-index is used _purely_ to set the correct mode (at 
least as far as I can tell). *Clickety-click* well, seems you had 
something different in mind on June 8th, 2005. "make sure that the full 
pathname is created". This, of course, is missing from the builtin 
merge-one-file (yet).

> Not exactly pretty. If this is something people want to do often, we'd 
> need to generate some porcelain for it.

I fully expect people to agree that this is not the way to go. I think 
cherry-picking is what you were after, Bill, correct?

Having said that, I find myself picking changes from a side-branch (with 
too many one-liner commits) with this command:

	git diff -R sidebranch file1 file2.. | git apply --index

Ciao,
Dscho

Re: How to pull only a few files from one branch to another?

[Bill Lear]

On Wednesday, January 24, 2007 at 11:16:35 (+0100) Johannes Schindelin writes:
>...
>I fully expect people to agree that this is not the way to go. I think 
>cherry-picking is what you were after, Bill, correct?

I can't answer confidently, but I think that this is correct.


Bill

Re: How to pull only a few files from one branch to another?

[Yann Dirson]

On Tue, Jan 23, 2007 at 08:11:38PM -0800, Linus Torvalds wrote:
> Git *very* fundamentally tracks project state, not file state. Which means 
> that you very much can NOT try to "merge a file". It is a senseless 
> operation in git, and in fact, any SCM that allows it pretty much is 
> doomed to be a total piece of sh*t (*).

> (*) And I'm not saying that just because git doesn't do it. It's much 
> more fundamental than that. Once you start doing per-file branching and 
> merging, you've basically screwed yourself, and you'll never be able to 
> work on the project as a "whole project" any more - you no longer have a 
> well-defined history that actually is the history of the whole project.

In fact, I came some time ago on a workflow which could be seen as
quite similar to this issue.

The problem was about merging a new upstream release in a local
branch, when both branches had heavy changes.  Indeed this tree was
not using GIT but CVS, with upstream tarballs imported on a branch - I
just asked myself how it could be best handled with GIT, and could
only come to the conclusion that something in GIT was still missing.

What was particular about this tree, is that we are several people
working on it, namely developpers taking care of the app, and me as
build manager taking care of the build mechanics as well as kernel+OS.
So the task was logically divided: one dev would merge the app, and I
would merge everything else.  As you see, it's far from one-file
merges, but the problem is quite similar.


The idea which I came up with was inspired by the "partial merge"
feature in PRCS, which I had never used or even understood the purpose
before that date.

Basically, it was that if some set of files could be merged somewhat
independently from the rest, and we don't want to get GIT attempt to
merge them again when finally merging the whole, a commit of a partial
merge would somehow record that some files had already been merged.
And I happenned to think that it could be sufficient to create a
commit that would not be a merge commit itself (since it is not a
full-project merge), but which would instead reference "subcommits"
for the relevant parts of the tree that were merged during that
iteration.

I realize that solution would not really be perfect, as there is
always some coupling between the code and build stuff in a project.
And sometimes we could want to get finer granularity than the file
level.  But that's so far my best guess at finding a solution.

How would you handle such a situation ?

Best regards,
-- 
Yann.

Re: How to pull only a few files from one branch to another?

[Jakub Narebski]

Yann Dirson wrote:

> What was particular about this tree, is that we are several people
> working on it, namely developpers taking care of the app, and me as
> build manager taking care of the build mechanics as well as kernel+OS.
> So the task was logically divided: one dev would merge the app, and I
> would merge everything else.  As you see, it's far from one-file
> merges, but the problem is quite similar.

This I think would be best handled by (future/prototype) submodules
support.


I wonder if fake-recording resolutions in git-rerere would help
in that case...
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: How to pull only a few files from one branch to another?

[Yann Dirson]

On Wed, Jan 24, 2007 at 09:46:49PM +0100, Jakub Narebski wrote:
> Yann Dirson wrote:
> 
> > What was particular about this tree, is that we are several people
> > working on it, namely developpers taking care of the app, and me as
> > build manager taking care of the build mechanics as well as kernel+OS.
> > So the task was logically divided: one dev would merge the app, and I
> > would merge everything else.  As you see, it's far from one-file
> > merges, but the problem is quite similar.
> 
> This I think would be best handled by (future/prototype) submodules
> support.

That could be used for some cases, but it's harder to use to avoid
getting a Makefile merged with the rest of the subtree.


> I wonder if fake-recording resolutions in git-rerere would help
> in that case...

Hm.  Looks like that would require being able to communicate rerere
results from one workspace to another.

Re: How to pull only a few files from one branch to another?

[Jakub Narebski]

Yann Dirson wrote:
> On Wed, Jan 24, 2007 at 09:46:49PM +0100, Jakub Narebski wrote:
>> Yann Dirson wrote:
>> 
>>> What was particular about this tree, is that we are several people
>>> working on it, namely developpers taking care of the app, and me as
>>> build manager taking care of the build mechanics as well as kernel+OS.
>>> So the task was logically divided: one dev would merge the app, and I
>>> would merge everything else.  As you see, it's far from one-file
>>> merges, but the problem is quite similar.

>> I wonder if fake-recording resolutions in git-rerere would help
>> in that case...
> 
> Hm.  Looks like that would require being able to communicate rerere
> results from one workspace to another.

Another solution would be to record partial merge (using "ours" strategy
for files we don't want to merge), then --amend it. 

-- 
Jakub Narebski
Poland

Re: How to pull only a few files from one branch to another?

[Linus Torvalds]

On Wed, 24 Jan 2007, Yann Dirson wrote:
> 
> What was particular about this tree, is that we are several people
> working on it, namely developpers taking care of the app, and me as
> build manager taking care of the build mechanics as well as kernel+OS.
> So the task was logically divided: one dev would merge the app, and I
> would merge everything else.  As you see, it's far from one-file
> merges, but the problem is quite similar.

Yes, it's basically the same thing.

And the sad part is that you can easily come up with tons of examples of 
where this makes sense, and indeed there are no fundamental problems at 
all as long as the "per-file" (or "file group" - the thing doesn't really 
change) history is strictly a perfect subset of the "whole project" 
history topology.

The reason it fundamentally doesn't work in the long run is that people 
invariably will then do things that do _not_ have the same "history 
topology". You'll have one file that has some of its history mixed in two 
different branches that _globally_ share none of that per-file history, so 
then the global history no longer matches the history of that file, and 
the "topology" of the history cannot be mapped from one to the other.

So what do I mean by that?

What git does is to track the history as it is shared by ALL files in the 
whole project (or, the way I prefer to think about it: it's not about 
independent files at all, it's _always_ about the whole collection).

This makes sense, because what does "history" actually mean? Forget about 
what it means as a word in English, and concentrate on what its meaning is 
strictly from a technical standpoint. Why does history matter? Why don't 
we just have a set of commits in date order?

>From a technical standpoint, the thing that makes history matter (and why 
"set of commits in date order" is useless) is because it gives us a COMMON 
ANCESTOR! And *that* is the only thing that matters.

Now, what is fundamentally wrong with doing per-file history? 

Now, if you've followed this argument, you should go "Aaahh! Obvious!".

The thing that is _fundamentally_ wrong with per-file history is that it 
breaks the one AND ONLY point of having history in the first place. There 
is no well-defined "common ancestor" notion on a repository level any 
more. You've made the history meaningless on a repo level, and it's now 
only meaningful on a file level.

So whats' wrong with that? You could still do merges, file by file, and 
just do the common ancestry that way, couldn't you?

Yes you can. It's just that if you do, you're a total moron, and you're 
back to the dark ages and playing with CVS (and possibly SVN - it's too 
early to say yet, since SVN can't do merges and history AT ALL right now 
as far as I know - it has the same merge logic as CVS has, I think, 
which is to say that it doesn't really understand the point of having 
history in the first place).

Which gets me back to where I started: you could (at least in in theory)
allow a situation where you allowed file-level merges AS LONG AS THEY 
NEVER CLASH WITH THE GLOBAL HISTORY.

In other words, I could imagine starting a branch for a particular file 
(and when I say "file", it can be any arbitrary subset of the full state), 
and having file-level history at that level, but you must NEVER then merge 
that file across into _another_ global branch, because if you do, then the 
"global history" has lost all value, since it's no longer actually global 
history any more.

So you have to choose: do you want to track things file by file, or do you 
want to track the whole project. You literally can't say "I want to do 
both!". Because tracking history file by file breaks the whole concept of 
tracking global history, the moment you start merging individual files 
across branch boundaries.

> The idea which I came up with was inspired by the "partial merge"
> feature in PRCS, which I had never used or even understood the purpose
> before that date.

I really don't think that people understand how fundamentally broken the 
whole concept is.

I think git should help you combine partial state between different 
branches, but you should always realize that IT IS NOT A MERGE! It's more 
like a "cherry-pick" - except you cherry-pick multiple commits "in space" 
instead of cherry-picking one commit "in time".

(The way I personally view git, "space" is the workspace - aka contents - 
while "time" is the history, aka the relationship of commits")

So it's perfectly ok to take data from other branches and include them in 
your current one. Nobody doubts that, and indeed, some forms of it we 
already have nice tools to help you do that ("cherry-pick" in particular).

And yes, doing a simple

	git diff commit..othercommit filename | 
		git-apply --index && git commit

is really just a way to "cherry-pick" the data when it's located in 
"space" instead of in "time" (ie we restrict it to a particular region of 
the workspace, and cherry-pick the work we did over a long time: this is 
100% equivalent to "git cherry-pick", which does it the other way around: 
it cherry-picks the work restricted "in time", but unstrstricted "in 
space").

But when you do this YOU MUST NOT CALL IT A MERGE!

Because you _by_definition_ don't actually do the ONE thing that is the 
whole point of a merge: the result does not become a common ancestor of 
the result.

So when you cherry-pick, you don't merge: you just create a bug-standard 
commit. It may contain data from another branch, but it is NOT a merge of 
the other branch.

> How would you handle such a situation ?

See above. Just create them as individual commits, and perhaps point to 
where the data came from in the commit comment, but don't try to think 
they are merges.

Btw, to explain my point perhaps even more fully, let me give an example 
of the "reverse" situation:

	git merge -s ours

This actually IS a merge, even though we don't actually make ANY CHANGES 
AT ALL to the tree, and we don't take any actual data at all from the 
other branch, and just state: "the result of the merge is always the 
current branch contents".

So why is it a merge? Exactly for the same reason that a partial-file 
"merge" is NOT a merge. It's a merge because the whole point of the "ours" 
strategy is to say "This is now going to be the common ancestor for these 
two states going forward". And that is the _definition_ of a merge, since 
that's the whole (and ONLY) point of having history in the first place.

So it all really boils down to a simple question: "What is the meaning of 
'history'?"

Once you understand what history is all about, you understand why a 
"merge" of a few files isn't actually a merge at all, but just a 
cherry-pick.

(And btw, that doesn't mean that we wouldn't use the "merge" program to do 
it. The way we actually implement "cherry-pick" is to physically do a 
"merge" of the contents, because passing patches forwards and backwards is 
just stupid. So the issue is somewhat confused by the different "levels" 
of meaning of the word "merge". There's the pure "content merge", which is 
a purely technical method of combining data, and which is usually just a 
three-way RCS-merge. And then there is the "merge commit", which tells you 
something about HISTORY, which is a lot more fundamental and a lot more 
important).

You can often do the "data merge" by hand. It's the least of your 
problems, and it's not even very interesting. The _interesting_ part of a 
git merge is what it means for history!

Doing cherry-picking (whichever kind you want to) is fine, but you have to 
realize that it also means that you will have a potentially harder time to 
do a "true merge" of the two branches later. Once your per-file history 
doesn't match the whole-project history, you end up losing a lot of the 
automatic goodness of simple merges, and you get back to the old CVS merge 
hell.

But the good news is that at least it won't be any _worse_ than what a lot 
of people have long since learnt about CVS merging. It will be "hell" only 
when compared to what you can do when you don't play games, and when you 
merge nicely.

			Linus

Re: How to pull only a few files from one branch to another?

[Daniel Barkalow]

On Tue, 23 Jan 2007, Bill Lear wrote:

> I have a long-running topic branch.  I have fixed a few nits on
> the master branch that I would like on the topic branch.  How do I
> pull in only a few files from the head of the master branch?

You don't pull in a few files, you apply the changes made in a few 
commits:

  git diff HEX_OF_NIT_FIX^ HEX_OF_NIT_FIX | git apply

If there's other stuff in the nit-fixing commit, shame on you, but you can 
edit the patch before applying it to remove everything that's not what you 
want.

(Incidentally, I think "git diff ^ {commit}" should be made to do "git 
diff {commit}^ {commit}"; i.e., if there is a single other revision 
provided, interpret a modifier not applied to anything as applying to that 
revision, in the "what else could that possibly mean?" department.)

	-Daniel
*This .sig left intentionally blank*

Re: How to pull only a few files from one branch to another?

[Bill Lear]

On Wednesday, January 24, 2007 at 00:29:00 (-0500) Daniel Barkalow writes:
>On Tue, 23 Jan 2007, Bill Lear wrote:
>
>> I have a long-running topic branch.  I have fixed a few nits on
>> the master branch that I would like on the topic branch.  How do I
>> pull in only a few files from the head of the master branch?
>
>You don't pull in a few files, you apply the changes made in a few 
>commits:
>
>  git diff HEX_OF_NIT_FIX^ HEX_OF_NIT_FIX | git apply
>...

Good technique.  Thank you for sharing.

I had been thinking about trying something along the lines of Junio's
"Separating topic branches" posted on the "GIT Howto Index" page.  I
may have tried that had I been braver.

I probably should have thought ahead and made this fix on a branch,
merged it into my master branch and then into my topic branch ... I
think.


Bill

Re: How to pull only a few files from one branch to another?

[Shawn O. Pearce]

Bill Lear <rael@zopyra.com> wrote:
> I probably should have thought ahead and made this fix on a branch,
> merged it into my master branch and then into my topic branch ... I
> think.

Yes.  That is standard "good practice" with Git.  Unfortunately it
does require a little planning ahead if the base to start on isn't
immediately obvious (you need to think "I want this here, and here,
and here.. and compute the merge base").

-- 
Shawn.

Re: How to pull only a few files from one branch to another?

[Andy Whitcroft]

Bill Lear wrote:
> On Wednesday, January 24, 2007 at 00:29:00 (-0500) Daniel Barkalow writes:
>> On Tue, 23 Jan 2007, Bill Lear wrote:
>>
>>> I have a long-running topic branch.  I have fixed a few nits on
>>> the master branch that I would like on the topic branch.  How do I
>>> pull in only a few files from the head of the master branch?
>> You don't pull in a few files, you apply the changes made in a few 
>> commits:
>>
>>  git diff HEX_OF_NIT_FIX^ HEX_OF_NIT_FIX | git apply
>> ...
> 
> Good technique.  Thank you for sharing.
> 
> I had been thinking about trying something along the lines of Junio's
> "Separating topic branches" posted on the "GIT Howto Index" page.  I
> may have tried that had I been braver.
> 
> I probably should have thought ahead and made this fix on a branch,
> merged it into my master branch and then into my topic branch ... I
> think.

Perhaps you can just rebase that one fix 'back' onto the base.  Then
merge it into both.

-apw

Re: How to pull only a few files from one branch to another?

[Junio C Hamano]

Daniel Barkalow <barkalow@iabervon.org> writes:

> You don't pull in a few files, you apply the changes made in a few 
> commits:
>
>   git diff HEX_OF_NIT_FIX^ HEX_OF_NIT_FIX | git apply
>
> If there's other stuff in the nit-fixing commit, shame on you, but you can 
> edit the patch before applying it to remove everything that's not what you 
> want.

Or just:

	git cherry-pick HEX_OF_NIT_FIX

possibly followed by tweaking the result and running:

	git commit --amend

Re: How to pull only a few files from one branch to another?

[Jakub Narebski]

Daniel Barkalow wrote:

> (Incidentally, I think "git diff ^ {commit}" should be made to do "git 
> diff {commit}^ {commit}"; i.e., if there is a single other revision 
> provided, interpret a modifier not applied to anything as applying to that 
> revision, in the "what else could that possibly mean?" department.)

"git diff <commit>^!" doesn't work?
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: How to pull only a few files from one branch to another?

[Daniel Barkalow]

On Wed, 24 Jan 2007, Jakub Narebski wrote:

> Daniel Barkalow wrote:
> 
> > (Incidentally, I think "git diff ^ {commit}" should be made to do "git 
> > diff {commit}^ {commit}"; i.e., if there is a single other revision 
> > provided, interpret a modifier not applied to anything as applying to that 
> > revision, in the "what else could that possibly mean?" department.)
> 
> "git diff <commit>^!" doesn't work?

It generates the right result, but it hasn't become something I 
automatically think of. :) For that matter, "git diff <commit>{^,}" also 
works, but, again, I don't think of it instantly.

	-Daniel
*This .sig left intentionally blank*