Merges without bases

Merges without bases

[Darrin Thompson]

I have a weird situation I want to support. I want to be able to merge a
foreign-tree repeatedly.

What makes the foreign tree foreign is that it may not yet share any
history with this branch.

What I tried to do was fetch foreign tree material and run octopus.
Octopus understandably barfed when it could not find a merge base for
the foreign tree.

What I think is newsworthy is that I figured out a sick way to get it
done. I created an empty tree object and used it as the merge base for
git-read-tree -m.

Could git-read-tree -m 3-args be made smart enough to treat a 0 as arg 1
as an implicit empty tree?

Once that is done, git octopus will be able to handle the no merge base
case.

--
Darrin

Re: Merges without bases

[Darrin Thompson]

That didn't come out clearly. Restating:

On Thu, 2005-08-25 at 16:10 -0500, Darrin Thompson wrote:
> Could git-read-tree -m 3-args be made smart enough to treat a 0 as arg 1
> as an implicit empty tree?
> 

Could git-read-tree -m treat an argument of "0" as an implicit empty
tree? It mainly seems useful as the first arg in the three arg -m form.

> Once that is done, git octopus will be able to handle the no merge base
> case.
> 

Once that is done, git octopus could _optionally_ handle the "I can't
find a merge base" case by passing 0 as the merge base to git-read-tree.

--
Darrin

Re: Merges without bases

[Junio C Hamano]

Darrin Thompson <darrint@progeny.com> writes:

> I have a weird situation I want to support. I want to be able to merge a
> foreign-tree repeatedly.
>
> What makes the foreign tree foreign is that it may not yet share any
> history with this branch.

I believe that's exactly what Linus did when he merged gitk into
git.  As you discovered, the initial one could easily be
scripted, if you wanted to, with something like this:

        #!/bin/sh
        #
        # git-merge-projects-script
        #
        . git-sh-setup-script || die "Not a git archive."

        foreign_project="$1"
        head=`git-rev-parse --verify HEAD` &&
        foreign=`git-rev-parse --verify $foreign_project^0` || exit

        rm -f .no-such-file
        empty=`GIT_INDEX_FILE=.no-such-file git-write-tree`
        git-read-tree -m -u $empty $head $foreign ||
        git-merge-cache -o git-merge-one-file-script -a || exit

        tree=`git-write-tree` &&
        echo Merge $foreign_project in. |
                git-commit-tree $tree -p $head -p $foreign \
                >"$GIT_DIR/HEAD" &&
	git show-branch --more=10 HEAD $foreign_project

Unlike my other "scripts written in e-mail buffer", I
actually tested this one ;-).

	$ cd /var/tmp && rm -fr junk && mkdir junk
	$ cd /var/tmp/junk
	$ git init-db
        $ cat >./git-merge-projects-script
        : type the above and end with ^D
	$ chmod +x git-merge-projects-script
        $ git add git-merge-projects-script
	$ git commit -m 'My Project'
	$ git fetch http://www.kernel.org/pub/scm/git/git.git master:git
	$ ./git-merge-projects-script git
	$ git diff git..HEAD

The "weird" situation to cause "git resolve" barf happens only
for the first time and once they are merged you can repeatedly
pull from that subset foreign branch without any "weirdo"
support.  Since even the oddball initial case can easily be
scripted like above, and that initial case should happen very
very rarely anyway, I do not think this deserves any core-level
change, such as changes to read-tree you suggest.

The above merge-projects-script _may_ deserve a "contrib"
status to be in the source tree, though.

One thing that makes me reluctant to recommend this "merging
unrelated projects" business is that I suspect that it makes
things _much_ harder for the upstream project that is being
merged, and should not be done without prior arrangement; Linus
merged gitk after talking with paulus, so that was OK.

Suppose the above "My Project" is published, people send patches
for core GIT part to it, and you as the maintainer of that "My
Project" accept those patches.  The users of "My Project" would
be happy with the new features and wouldn't care less where
their core GIT tools come from.  But how would _I_ pull from
that "My Project", if I did not want to pull unrelated stuff in?

In this particular example, it only has git-merge-projects
script which _is_ related to core GIT so this objection would
not apply, but if I were paulus (gitk maintainer whose project
has only one file, the great gitk script) and if the git.git
repository had a lot of changes to gitk, I would like to pick up
updates from there without pulling the rest of core GIT.  That
is not something the current set of tools support, and offhand I
do not think of a good way to implement it cleanly.  That is the
reason why I never pick up any patch for gitk myself --- I
always slurp changes to gitk via paulus tree, or feed patches
to him and then slurp the changes from him.

What I think _might_ deserve a bit more support would be a merge
of a foreign project as a subdirectory of a project.  Linus
could have made gitk/ subdirectory in the core git project, and
made that subtree in sync with the toplevel of paulus gitk
project.  But even this could be done without the core-level
change.  You would run ls-tree of the main project to figure out
the tree object name of that subdirectory, and 3-way merge that
with the top of the paulus project tree, and replace the tree
entry with the resulting tree.

Once that kind of "merging an unrelated project as a
subdirectory of a superproject" workflow is established, pulling
from a subdirectory that corresponds to my project after my
project gets merged this way into the superproject would become
easier to manage and would even become a useful workflow
element.

Re: Merges without bases

[Darrin Thompson]

On Thu, 2005-08-25 at 15:26 -0700, Junio C Hamano wrote:
>         empty=`GIT_INDEX_FILE=.no-such-file git-write-tree`
>         git-read-tree -m -u $empty $head $foreign ||

ooooo. Tricky.

Thanks for the script. That's a bad, bad hack. :-)

> One thing that makes me reluctant to recommend this "merging
> unrelated projects" business is that I suspect that it makes
> things _much_ harder for the upstream project that is being
> merged, and should not be done without prior arrangement; Linus
> merged gitk after talking with paulus, so that was OK.
> 

What I'm going to do is actually an inversion of that. Publishing a
repository with the _intent_ of being merged into existing history, and
observing obvious naming conventions as the "prior arrangement".

I thought once I got the initial baseless merges done and committed that
I do fetch-octopus from that point on. But octopus was still complaining
about not finding a merge base. I'm going to verify that I didn't just
mess something up in the process.

If I can get octopus working as the tool for doing merges _after_ the
baseless merges then I can live with the current situation. 

--
Darrin

Re: Merges without bases

[Tim Ottinger]

Darrin Thompson wrote:

>What I'm going to do is actually an inversion of that. Publishing a
>repository with the _intent_ of being merged into existing history, and
>observing obvious naming conventions as the "prior arrangement".
>
>I thought once I got the initial baseless merges done and committed that
>I do fetch-octopus from that point on. But octopus was still complaining
>about not finding a merge base. I'm going to verify that I didn't just
>mess something up in the process.
>
>If I can get octopus working as the tool for doing merges _after_ the
>baseless merges then I can live with the current situation. 
>  
>

Heh.  Git repositories as components.

-- 
                             ><>
... either 'way ahead of the game, or 'way out in left field.

Re: Merges without bases

[Daniel Barkalow]

On Thu, 25 Aug 2005, Junio C Hamano wrote:

> One thing that makes me reluctant to recommend this "merging
> unrelated projects" business is that I suspect that it makes
> things _much_ harder for the upstream project that is being
> merged, and should not be done without prior arrangement; Linus
> merged gitk after talking with paulus, so that was OK.

I'd still like to revive my idea of having projects overlaid on each
other, where the commits in the project that absorbed the other project
say, essentially, "also include this other commit, but any changes to
those files belong to that branch, not this one". That way, Linus could
have included gitk in git, but changes to it, even when done in a git
working tree, would show up in commits that only include gitk. (git
actually can handle this with the alternative index file mechanism that
Linus mentioned in a different thread.)

Definitely post-1.0, of course.

> Suppose the above "My Project" is published, people send patches
> for core GIT part to it, and you as the maintainer of that "My
> Project" accept those patches.  The users of "My Project" would
> be happy with the new features and wouldn't care less where
> their core GIT tools come from.  But how would _I_ pull from
> that "My Project", if I did not want to pull unrelated stuff in?

With the right info, the tools could be made to automatically generate
suitable commits, because those files would be tracked by a separate index
file and committed into a separate branch, which would then be reincluded
(by reference) in the containing project.

	-Daniel
*This .sig left intentionally blank*

Re: Merges without bases

[Junio C Hamano]

Daniel Barkalow <barkalow@iabervon.org> writes:

> I'd still like to revive my idea of having projects overlaid on each
> other, where the commits in the project that absorbed the other project
> say, essentially, "also include this other commit, but any changes to
> those files belong to that branch, not this one". That way, Linus could
> have included gitk in git, but changes to it, even when done in a git
> working tree, would show up in commits that only include gitk. (git
> actually can handle this with the alternative index file mechanism that
> Linus mentioned in a different thread.)

Yes, I would love to see that cleanly done in a way that does not
confuse uninitiated (not being sarcastic at all.  Just cheering
up somebody with a better idea than I have --- I would be lost
if I were to be tasked to do it by Emperor Penguin himself or
somebody else ;-)).

Tonight I added another root in the git.git repository, but I
cheated.

> Definitely post-1.0, of course.

Agreed.

Re: Merges without bases

[Eric W. Biederman]

Junio C Hamano <junkio@cox.net> writes:

> Daniel Barkalow <barkalow@iabervon.org> writes:
>
>> I'd still like to revive my idea of having projects overlaid on each
>> other, where the commits in the project that absorbed the other project
>> say, essentially, "also include this other commit, but any changes to
>> those files belong to that branch, not this one". That way, Linus could
>> have included gitk in git, but changes to it, even when done in a git
>> working tree, would show up in commits that only include gitk. (git
>> actually can handle this with the alternative index file mechanism that
>> Linus mentioned in a different thread.)
>
> Yes, I would love to see that cleanly done in a way that does not
> confuse uninitiated (not being sarcastic at all.  Just cheering
> up somebody with a better idea than I have --- I would be lost
> if I were to be tasked to do it by Emperor Penguin himself or
> somebody else ;-)).

I think when it comes to simplicity it would be better to have 
something that would filter all of the changes on a branch
by pathname and create a branch against the original project
with just those changes.

Then we can do the noop merge of that branch into the larger
project, and we can merge that branch into the original project.

The nice part of doing it after the fact by just filtering changes
is you don't have to plan ahead to handle that case, which
should be a lot easier to handle.

The set of pathnames to filter could be easily stored in the .git
metadata so doing repeatedly is straight forward.

Eric

Re: Merges without bases

[Martin Langhoff]

On 8/26/05, Junio C Hamano <junkio@cox.net> wrote:
> their core GIT tools come from.  But how would _I_ pull from
> that "My Project", if I did not want to pull unrelated stuff in?

and then... 

> What I think _might_ deserve a bit more support would be a merge
> of a foreign project as a subdirectory of a project.  Linus

tla has an interesting implementation (and horrible name) for
something like this. In Arch-speak, they are called 'configurations',
a versioned control file that describes that in subdirectory foo we
import from this other repo#branch.

In cvs, you just do nested checkouts, and trust a `cvs update` done at
the top will do the right thing;  and in fact recent cvs versions do.

After using cvs and arch for a while, my opinion is that all this
stuff is _bad_, and you want a makefile that pulls the projects
together when you build them. Different projects are going to use
different SCMs anyway, and you'll have to live with how to tag a
release across repositories/scms, and I haven't seen any answer I
like.

cheers,


martin

Re: Merges without bases

[Daniel Barkalow]

On Fri, 26 Aug 2005, Martin Langhoff wrote:

> On 8/26/05, Junio C Hamano <junkio@cox.net> wrote:
> > their core GIT tools come from.  But how would _I_ pull from
> > that "My Project", if I did not want to pull unrelated stuff in?
>
> and then...
>
> > What I think _might_ deserve a bit more support would be a merge
> > of a foreign project as a subdirectory of a project.  Linus
>
> tla has an interesting implementation (and horrible name) for
> something like this. In Arch-speak, they are called 'configurations',
> a versioned control file that describes that in subdirectory foo we
> import from this other repo#branch.
>
> In cvs, you just do nested checkouts, and trust a `cvs update` done at
> the top will do the right thing;  and in fact recent cvs versions do.

The problem with both of these (and doing it in the build system) is that,
when a project includes another project, you generally don't want whatever
revision of the included project happens to be the latest; you want the
revision of the included project that the revision of the including
project you're looking at matches. That is, if App includes Lib, and
you're looking at an App commit, you want to have the version of Lib that
the commit was made with, not the latest version of Lib, which may not be
backwards compatible across non-release commits, or, in any case, won't
help in reconstructing a earlier state. I think a primary function of a
SCM is to be able to say, "It worked last Friday, and it's broken now.
What's different?" If the answer is, "On Saturday, we updated the
included Lib to their version from Thursday, which is broken", it'll be
really hard to track down without special tracking.

I think it's the lack of the special tracking, therefore, that makes this
not a good feature in most SCMs, and makes them not better than having the
build system do it (and potentially worse, if you've got your build system
checking out a version specified in a version-controlled file). But I
think that git can do better, including support for the required version
sometimes being a locally modified one and sometimes being the official
one when the local modifications have been accepted upstream.

	-Daniel
*This .sig left intentionally blank*

Re: Merges without bases

[Martin Langhoff]

On 8/27/05, Daniel Barkalow <barkalow@iabervon.org> wrote:
> The problem with both of these (and doing it in the build system) is that,
> when a project includes another project, you generally don't want whatever
> revision of the included project happens to be the latest; you want the
> revision of the included project that the revision of the including
> project you're looking at matches. That is, if App includes Lib, and

Exactly - so you do it on a tag, or a commit date with cvs. With Arch,
GIT and others that have a stable id for each commit, you can use that
or the more user-friendly tags.

The project pulling the libs has the makefile, and the makefile says
'pull library-foo revision xxx'. If a later revision yyy is known to
work well, you update the makefile and commit it. Perfectly version
controlled, no need for special purpose machinery ;)

The good thing here is that a makefile will know how to handle the
situation if the external lib is hosted in Arch, in SVN, or Visual
SourceSafe. If your external lib is only available as a tarball in a
url, you can fetch that and uncompress it too. Arch configurations are
_cute_ but useless in any but the most narrow cases.

I want my SCM to be a good SCM, but this kind of interop is better
left to general purpose languages. Letting the build system do it
seems to be 'best-practice' and the right thing to do.

cheers,


martin

Re: Merges without bases

[Daniel Barkalow]

On Sat, 27 Aug 2005, Martin Langhoff wrote:

> On 8/27/05, Daniel Barkalow <barkalow@iabervon.org> wrote:
> > The problem with both of these (and doing it in the build system) is that,
> > when a project includes another project, you generally don't want whatever
> > revision of the included project happens to be the latest; you want the
> > revision of the included project that the revision of the including
> > project you're looking at matches. That is, if App includes Lib, and
>
> Exactly - so you do it on a tag, or a commit date with cvs. With Arch,
> GIT and others that have a stable id for each commit, you can use that
> or the more user-friendly tags.

I'm thinking of cases like openssl, openssh, and libcrypto. Openssl and
openssh both use libcrypto but not each other (looking at the ldd output,
rather than packaging). However, it would be too much of a pain to work
directly on libcrypto without working through some other package, because
the library doesn't have its own applications. Furthermore, if you're
doing much to libcrypto, you're likely doing it in the context of a
particular application (say, for example, ssh needs a new cipher that
isn't supported for SSL at the time). You'd want to make simultaneous
changes to libcrypto to implement the new feature and to openssh to use
it; neither can be validated until the other is written, which means that
you'll have both projects checked out and dirty (in the cache sense) at
the same time, and be building the using project.

It would also be good to be able to check in this whole thing through the
version control system, rather than partially through a change to the
build system. That is, if I change the included libcrypto, commit it, and
commit the including openssh, the system as a whole should understand that
I want to change which commit of libcrypto gets used. Similarly, it would
be good to merge changes into the libcrypto used by openssh with the same
procedure used to merge changes to openssh itself, including supporting
non-fast-forward when there's a local version in use.

(Of course, currently, libcrypto is strictly part of openssl, because it
would be too much of a pain with the present version control to make it
independant, and openssh depends on openssl, despite not even linking
against -lssl, because openssl got libcrypto first.)

> The good thing here is that a makefile will know how to handle the
> situation if the external lib is hosted in Arch, in SVN, or Visual
> SourceSafe. If your external lib is only available as a tarball in a
> url, you can fetch that and uncompress it too. Arch configurations are
> _cute_ but useless in any but the most narrow cases.

Certainly, if it's sufficiently external to be in a different SCM it
should be handled by the build system. Actually, if it's even nearly that
external, it's probably going to be handled best by requiring people to go
get it themselves.

I find it odd that you say that the standard approach is to have the build
system fetch a version of the included package; my experience is that
projects either just report (or fail to report) a dependancy on having the
other package or they copy the project into their project. The former
means they can't change it (which is generally good, unless it becomes
necessary), while the latter causes update problems (c.f. zlib).

I think that Arch configurations and the CVS equivalent are, in fact,
useless, but that this is only due to implementation being insufficiently
clever, not due to the concept being inherently bad; I feel the same way
about distributed development under Arch, which is really nice under git,
so I have hope that something better could be done.

	-Daniel
*This .sig left intentionally blank*