Mixing and matching multiple projects

Mixing and matching multiple projects

[Junio C Hamano]

Imagine that there is an open source embedded appliance project "xyzzy"
that consists of two submodules and the top-level superproject that binds
them and serves as the scaffolding, perhaps organized like this:

    /s/xyzzy/.git       -- the superproject
    /s/xyzzy/os/.git    -- the "os" submodule
    /s/xyzzy/u/.git     -- the "userspace" submodule

The "os" submodule may contain various subdirectories, among which there
may be a device drivers directory.  Further imagine that the Makefile in
that directory, /s/xyzzy/os/drivers/Makefile, is written in such a way
that it can run make in any subdirectory of drivers/ and can link with
resulting *.o files in these subdirectories.

Now suppose somebody wanted to add an add-on device driver for frotz
device that does not appear in the open source tree.  Further suppose that
the open source folks do not want to accept support for "frotz" driver in
their tree for whatever reason.

At the working tree level, he can add /s/xyzzy/os/drivers/frotz/ directory
and populate its sources there. Everything else is already set up to build
and link to it properly. But how would he add that directory to git?

He could fork the /s/xyzzy/os/.git submodule and add "drivers/frotz/"
directory, and maintain his own fork of the submodule, which in turn means
that he has to fork the top-level superproject /s/xyzzy/.git because its
tree has to point at a commit in /s/xyzzy/os/.git repository that has the
"drivers/frotz" directory in it, which is different from the open source
version. A proprietary fork has to happen somewhere; that is a given.

But I wonder if we can do without forking the /s/xyzzy/os/.git submodule?

If we *could* do this:

	cd /s/xyzzy
        mkdir os/drivers/frotz && populate the directory with sources
        git add os/drivers/frotz ;# to the top-level superproject!!

to add the "frotz driver" submodule directly to the superproject, then we
could leave /s/xyzzy/os/.git intact, letting it follow the open source
world. Because the superproject must be forked in order to keep track of
what happens in the appliance that supports the "frotz" driver anyway,
this could result in the minimum amount of forking from the end user's
point of view.

People who know the Git data structure of course can immediately see that
this is not supported (and I strongly suspect unsupportable, but my
suspicion has a history of being wrong from time to time, and that is why
I am sending this message).  The index at /s/xyzzy/.git/index has to be
able to record "os" as a gitlink and "os/drivers/frotz/frotz.c" as a
regular blob at the same time, which is a D/F conflict.

Even if you modify the index D/F conflict check to allow this, you cannot
write the top-level tree object for the superproject, as "os" needs to be
recorded as a gitlink to bind the /s/xyzzy/os/.git submodule, while you
also have to have another "os" in the same top-level tree object as a tree
object that has a single entry "drivers" (which is a tree) in it, under
which all the "os/drivers/frotz" subdirectory hang.  All tree traversal
code would be very unhappy to see such a tree with "duplicate" entry,
including the unpack-trees machinery used for merging and switching
branches, connectivity machinery used for fsck, object enumeration, and
object transfer.  I imagine that it *could* be done, but it won't be a
mere two-weekend hack; it has to first destroy pretty much everything in
the current codebase and rebuild it to add such a support.

The "frotz" submodule could be added as /s/xyzzy/frotz/.git and the build
infrastructure at /s/xyzzy/.git (which we are going to fork anyway) could
be tweaked so that it creates a symbolic link in /s/xyzzy/os/drivers to
point there, pretending that there is a 'frotz' subdirectory. While that
would certainly be a workaround, I am wondering if people who have used
submodules and/or nested projects more than I have better solutions to
this puzzle.

Comments?

Re: Mixing and matching multiple projects

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> But I wonder if we can do without forking the /s/xyzzy/os/.git submodule?
>
> If we *could* do this:
>
> 	cd /s/xyzzy
>       mkdir os/drivers/frotz && populate the directory with sources

A crucial step was missing here for my story to make *any* sense. Sorry.
A step to create a repository at /s/xyzzy/os/drivers/frotz/.git must be
here, like this:

        (cd os/drivers/frotz && git init && git add . && git commit)

And then that is added as a submodule to the superproject.

>       git add os/drivers/frotz ;# to the top-level superproject!!
>
> to add the "frotz driver" submodule directly to the superproject, then we
> could leave /s/xyzzy/os/.git intact, letting it follow the open source
> world. Because the superproject must be forked in order to keep track of
> what happens in the appliance that supports the "frotz" driver anyway,
> this could result in the minimum amount of forking from the end user's
> point of view.
>
> People who know the Git data structure of course can immediately see that
> this is not supported (and I strongly suspect unsupportable, but my
> suspicion has a history of being wrong from time to time, and that is why
> I am sending this message).  The index at /s/xyzzy/.git/index has to be
> able to record "os" as a gitlink and "os/drivers/frotz/frotz.c" as a
> regular blob at the same time, which is a D/F conflict.
>
> Even if you modify the index D/F conflict check to allow this, you cannot
> write the top-level tree object for the superproject, as "os" needs to be
> recorded as a gitlink to bind the /s/xyzzy/os/.git submodule, while you
> also have to have another "os" in the same top-level tree object as a tree
> object that has a single entry "drivers" (which is a tree) in it, under
> which all the "os/drivers/frotz" subdirectory hang.  All tree traversal
> code would be very unhappy to see such a tree with "duplicate" entry,
> including the unpack-trees machinery used for merging and switching
> branches, connectivity machinery used for fsck, object enumeration, and
> object transfer.  I imagine that it *could* be done, but it won't be a
> mere two-weekend hack; it has to first destroy pretty much everything in
> the current codebase and rebuild it to add such a support.
>
> The "frotz" submodule could be added as /s/xyzzy/frotz/.git and the build
> infrastructure at /s/xyzzy/.git (which we are going to fork anyway) could
> be tweaked so that it creates a symbolic link in /s/xyzzy/os/drivers to
> point there, pretending that there is a 'frotz' subdirectory. While that
> would certainly be a workaround, I am wondering if people who have used
> submodules and/or nested projects more than I have better solutions to
> this puzzle.
>
> Comments?

RE: Mixing and matching multiple projects

[Holding, Lawrence]

Junio C Hamano <gitster@pobox.com> writes:
> Junio C Hamano <gitster@pobox.com> writes:
> 
> > But I wonder if we can do without forking the /s/xyzzy/os/.git
submodule?
> >
> > If we *could* do this:
> >
> > 	cd /s/xyzzy
> >       mkdir os/drivers/frotz && populate the directory with sources
> 
> A crucial step was missing here for my story to make *any* sense.
Sorry.
> A step to create a repository at /s/xyzzy/os/drivers/frotz/.git must
be
> here, like this:
> 
>         (cd os/drivers/frotz && git init && git add . && git commit)
> 
> And then that is added as a submodule to the superproject.
> 
> >       git add os/drivers/frotz ;# to the top-level superproject!!
> >
> > to add the "frotz driver" submodule directly to the superproject,
then we
> > could leave /s/xyzzy/os/.git intact, letting it follow the open
source
> > world. Because the superproject must be forked in order to keep
track of
> > what happens in the appliance that supports the "frotz" driver
anyway,
> > this could result in the minimum amount of forking from the end
user's
> > point of view.
> >
> ........
> >
How about 
- creating a "driver" project in the superproject, 
- add a .gitignore or .git/exclude entry for the
/s/xyzzy/os/drivers/frotz folder into the os project
- symlinking the /s/xyzzy/os/drivers/frotz folder to the
/s/xyzzy/driver/frotz folder

some of this can be automated in the post checkout script of the
superproject.

Re: Mixing and matching multiple projects

[Junio C Hamano]

"Holding, Lawrence" <Lawrence.Holding@cubic.com> writes:

> How about 
> - creating a "driver" project in the superproject, 
> - add a .gitignore or .git/exclude entry for the
> /s/xyzzy/os/drivers/frotz folder into the os project
> - symlinking the /s/xyzzy/os/drivers/frotz folder to the
> /s/xyzzy/driver/frotz folder

Thanks.

I think that is essentially the same as what I wrote with "symlink" in my
last paragraph as a known-to-work workaround.  The real location of the
frotz driver, whether it is in a directory directly in the superproject,
or a separate submodule that is bound to the superproject outside the "os"
submodule, does not make an essential difference, as long as it is outside
"os".

I was asking if people who have used submodules have better solutions than
that "symlink" solution.

Re: Mixing and matching multiple projects

[Seth Robertson]

In message <7vhayptght.fsf@alter.siamese.dyndns.org>, Junio C Hamano writes:

        /s/xyzzy/.git       -- the superproject
        /s/xyzzy/os/.git    -- the "os" submodule
        /s/xyzzy/u/.git     -- the "userspace" submodule

    At the working tree level, he can add /s/xyzzy/os/drivers/frotz/ directory
    and populate its sources there. Everything else is already set up to build
    and link to it properly. But how would he add that directory to git?

    But I wonder if we can do without forking the /s/xyzzy/os/.git submodule?

    I am wondering if people who have used submodules and/or nested
    projects more than I have better solutions to this puzzle.

Using gitslave [http://gitslave.sf.net] instead of git-submodules, you
can *almost* get there.

Specifically, gitslave allows the superproject /s/xyzzy to attach a
subproject at /s/xyzzy/os/drivers/frotz (either recursively through
/os which you don't want to do or directly).

However, the problem is the .gitignore in the os subproject.  If we
didn't have some way to ignore drivers/frotz, various git commands
would get annoyed.  Adding it in the os subproject requires forking
which you said you didn't want to do.  However, you could use
core.excludesfile in the os subproject to ignore the necessary file.

gitslave doesn't support automatically using core.excludesfile (and
indeed would automatically make an entry in os/.gitignore when you set
up the hierarchy).  However, it isn't a lot of work to revert that
.gitignore commit and set up core.excludesfile; but setting up
core.excludesfile on every clone might get annoying.  It probably
wouldn't be too difficult to tearch gitslave how to do that
automagically.

Is using this third party tool and making the necessary changes or
manually doing the work after every clone worth it?  Do you need the
git-submodule semantics of freezing the subprojects at a particular
SHA instead of floating at the tip of a branch?  Are the branches used
between xyzzy and os desynchronized?  Only you can judge.  Actually it
seems possible that you could use gitslave for os/drivers/frotz and
git-submodules for os.  Whether that is wise…

I would be open to moving gitslave into contrib similar to what
git-subtree is under the process of doing, but gitslave would
certainly need some cleanup since it grew rather organically.

					-Seth Robertson