Project organisation and structure

Project organisation and structure

[k wayne]

Hello,
I'm new to git, and I hope this is the right mailinglist for what I'm
going to ask; it was the only one I found on http://git.or.cz. Sorry
if it's not or my question is remarkably stupid. I've read the git
documentation, of course, but it wasn't always easy to wrap my head
around all the concepts, so I might have missed an obvious solution to
my problem.

Currently, I've got a tree of self-written C++ source code for which I
don't use any SCM, and I'd like to start using git on it. However,
I've run into issues with my directory layout I cannot resolve on my
own.
I have a collection of libraries and programs, which aren't coupled
too much, but aren't really independent either; for example, library X
depends on Y, program Z on lib A and so forth. Since all of these libs
and programs are logically connected to each other in some way, I
would like to have them all in one central repository.
So I heard that I could create kind of a "meta-repository" for the
whole project, and have submodules for each library or program, which
sounds exactly like what I want. However, I've been told that my
current directory layout will not work this way with git.

This is how my project root directory (say it's ~) looks like:
~/build - Build files which apply for the whole project, like doxygen
build files.
~/include/$submodule - Each library/program has an own directory here,
in which all header files go.
~/projects/$submodule - Files related to an individual project, like
makefiles etc.
~/src/$submodule - Like the above two, but for source files.
~/test/$submodule - Again, a directory for each library/program,
containing files for (unit)tests.
~/doc - Documentation files for the project as a whole.
~/doc/$submodule - Documentation files for individual libraries/programs.

These are the directories I would like git to track for me. There are
some other dirs not listed here (e.g. for object files,) but I can
easily add them to .gitignore.

So, the git structure as I imagine it would look like this: ~/.git
contains the "meta-repository," in which all the submodules reside.
Each submodule would have its .git directory in ~/projects/$submodule.
Alternatively, ~/.git for the meta-repo and ~/.git-$subproject or
something like this would be okay too.

However, I have been told I cannot go to ~/projects/$submodule and do
a "git add ../../{include,src,test,doc}/$submodule" there. I could add
symlinks in ~/projects/$submodule to each of these dirs and add that
link, but this would not work with windows, I guess (would it work
with git, anyway?)

So, how can I manage my code without having to restructure my tree?

I'd like to keep it that way, because it is convenient to have only
one directory to pass to the compiler as an additional include dir,
and can include my header files as "submodule/someheader.hpp" which
is, in my opinion, helpful in organizing the code.

If there should be no way I can keep my directory setup the way it is
with the current git, would a feature request be a good idea (and
possibly implemented), or does my directory tree have a serious flaw I
haven't stumbled across yet, which makes it unusable anyways?

Thank you in advance.

Re: Project organisation and structure

[Andreas Ericsson]

k wayne wrote:
> Hello,
> I'm new to git, and I hope this is the right mailinglist for what I'm
> going to ask; it was the only one I found on http://git.or.cz. Sorry
> if it's not or my question is remarkably stupid. I've read the git
> documentation, of course, but it wasn't always easy to wrap my head
> around all the concepts, so I might have missed an obvious solution to
> my problem.
> 

This is the right place, and no you didn't miss anything obvious.


> I have a collection of libraries and programs, which aren't coupled
> too much, but aren't really independent either; for example, library X
> depends on Y, program Z on lib A and so forth. Since all of these libs
> and programs are logically connected to each other in some way, I
> would like to have them all in one central repository.
> So I heard that I could create kind of a "meta-repository" for the
> whole project, and have submodules for each library or program, which
> sounds exactly like what I want. However, I've been told that my
> current directory layout will not work this way with git.
> 
> This is how my project root directory (say it's ~) looks like:
> ~/build - Build files which apply for the whole project, like doxygen
> build files.
> ~/include/$submodule - Each library/program has an own directory here,
> in which all header files go.
> ~/projects/$submodule - Files related to an individual project, like
> makefiles etc.
> ~/src/$submodule - Like the above two, but for source files.
> ~/test/$submodule - Again, a directory for each library/program,
> containing files for (unit)tests.
> ~/doc - Documentation files for the project as a whole.
> ~/doc/$submodule - Documentation files for individual libraries/programs.
> 
> These are the directories I would like git to track for me. There are
> some other dirs not listed here (e.g. for object files,) but I can
> easily add them to .gitignore.
> 
> So, the git structure as I imagine it would look like this: ~/.git
> contains the "meta-repository," in which all the submodules reside.
> Each submodule would have its .git directory in ~/projects/$submodule.
> Alternatively, ~/.git for the meta-repo and ~/.git-$subproject or
> something like this would be okay too.
> 
> However, I have been told I cannot go to ~/projects/$submodule and do
> a "git add ../../{include,src,test,doc}/$submodule" there.

Right. That would break things rather horribly as you're trying to add
files below the root of your repository.

> I could add
> symlinks in ~/projects/$submodule to each of these dirs and add that
> link, but this would not work with windows, I guess (would it work
> with git, anyway?)
> 

It would work with git, and it will work on windows, sort of. Git knows
that the symlink is a symlink, but it will check out a proper copy of
the actual file in its place on windows. You just have to be careful
when updating it, as tests and stuff before you've committed to git
might break with one file being the old revision still.

> So, how can I manage my code without having to restructure my tree?
> 

Short answer; Lots of symlinks.

Longer answer:
Are you really sure that's the best structure for your code? I usually
find it absolutely simplest to bundle tests, buildrules and source
all in one directory (possibly with tests and documentation in a subdir),
but otherwise keep it all jumbled together. There's a very important
reason for this, and it's that your .cpp files shouldn't have to know
what some other buildsystem is doing with its header files, so you
should always be able to #include "header_for_this_api.hpp" and be
done with it.

If I were you, I'd do something like this:
/: top-level buildstuff
/doc: collection-wide documentation ("this suite of programs blahblah")
/t: integration tests
/lib/$library_subproject - holds library source/build/unit-tests/doc
/src/$program_subproject - holds program source/build/unit-tests/doc

After all, you're mostly updating one library or program at a time, right?
If you're not and all the API's are in wild flux, you should probably
just keep it jumbled in a single git repository for the moment and split
them later when things have calmed down a bit.

> I'd like to keep it that way, because it is convenient to have only
> one directory to pass to the compiler as an additional include dir,
> and can include my header files as "submodule/someheader.hpp" which
> is, in my opinion, helpful in organizing the code.
> 

With the layout above, you just pass -I./lib as include dir and use
	#include <$library/header.hpp>
in your cpp files, which is presumably what you'd use if you'd installed
the headers in the normal location too.

> If there should be no way I can keep my directory setup the way it is
> with the current git, would a feature request be a good idea (and
> possibly implemented),

A feature request to be able to track files outside the repository would
require some major surgery to the most basic of git objects, so I assume
such a feature wouldn't be too well received.

> or does my directory tree have a serious flaw I
> haven't stumbled across yet, which makes it unusable anyways?
> 

I'd call it unusable to have to hop to several different directories to
add a single public function to a library, but perhaps that's just me?

Perhaps your programs contain mini-libraries that are shared at link-time
between applications? My advice in that case is to turn them into libraries
and make proper API's from them. That way the above directory structure
will give you everything you want while maintaining each individual
component separated in its own repository.

http://people.redhat.com/drepper/goodpractice.pdf holds a very nice and
concise writeup on how to create API's. I follow to the letter in most
of my library-like projects.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: Project organisation and structure

[Boaz Harrosh]

k wayne wrote:
> Hello,
> I'm new to git, and I hope this is the right mailinglist for what I'm
> going to ask; it was the only one I found on http://git.or.cz. Sorry
> if it's not or my question is remarkably stupid. I've read the git
> documentation, of course, but it wasn't always easy to wrap my head
> around all the concepts, so I might have missed an obvious solution to
> my problem.
> 
> Currently, I've got a tree of self-written C++ source code for which I
> don't use any SCM, and I'd like to start using git on it. However,
> I've run into issues with my directory layout I cannot resolve on my
> own.
> I have a collection of libraries and programs, which aren't coupled
> too much, but aren't really independent either; for example, library X
> depends on Y, program Z on lib A and so forth. Since all of these libs
> and programs are logically connected to each other in some way, I
> would like to have them all in one central repository.
> So I heard that I could create kind of a "meta-repository" for the
> whole project, and have submodules for each library or program, which
> sounds exactly like what I want. However, I've been told that my
> current directory layout will not work this way with git.
> 
> This is how my project root directory (say it's ~) looks like:
> ~/build - Build files which apply for the whole project, like doxygen
> build files.
> ~/include/$submodule - Each library/program has an own directory here,
> in which all header files go.
> ~/projects/$submodule - Files related to an individual project, like
> makefiles etc.
> ~/src/$submodule - Like the above two, but for source files.
> ~/test/$submodule - Again, a directory for each library/program,
> containing files for (unit)tests.
> ~/doc - Documentation files for the project as a whole.
> ~/doc/$submodule - Documentation files for individual libraries/programs.
> 
> These are the directories I would like git to track for me. There are
> some other dirs not listed here (e.g. for object files,) but I can
> easily add them to .gitignore.
> 
> So, the git structure as I imagine it would look like this: ~/.git
> contains the "meta-repository," in which all the submodules reside.
> Each submodule would have its .git directory in ~/projects/$submodule.
> Alternatively, ~/.git for the meta-repo and ~/.git-$subproject or
> something like this would be okay too.
> 
> However, I have been told I cannot go to ~/projects/$submodule and do
> a "git add ../../{include,src,test,doc}/$submodule" there. I could add
> symlinks in ~/projects/$submodule to each of these dirs and add that
> link, but this would not work with windows, I guess (would it work
> with git, anyway?)
> 
> So, how can I manage my code without having to restructure my tree?
> 
> I'd like to keep it that way, because it is convenient to have only
> one directory to pass to the compiler as an additional include dir,
> and can include my header files as "submodule/someheader.hpp" which
> is, in my opinion, helpful in organizing the code.
> 
> If there should be no way I can keep my directory setup the way it is
> with the current git, would a feature request be a good idea (and
> possibly implemented), or does my directory tree have a serious flaw I
> haven't stumbled across yet, which makes it unusable anyways?
> 
> Thank you in advance.
> --

Your directory structure is common and intuitive in the sense that thats
what happens, you start coding then add stuff along the way. You don't
know from the start what will the tree look like.

Git is well suited for your tree and will eat it with out solt

Do not use subprojects these are used with two separately developed
projects that are dependent (one on the other not two ways).
For example I have a GUI app that uses wxWidgets. Instead of instructing
my users on where to get it and what exact version to use, with what 
version of my program, I mirror it in a submodule and this is all done
automatically for me and my cloners.

What you need is topic branches. Not all branches need have all the
same files. In fact they can have nothing in common. The only thing
with git is that at a certain checkout in time only these files will
be present on disk.

So lets say you have:

/include/lib4foobar
/include/lib4bar
/src/lib4foobar
/src/lib4bar
/src/foo
/src/bar

Lets complicate it and say lib4bar is also dependent on lib4foobar
then you have these branches: lib4foobar lib4bar foo bar

When you work on a library/app you take your foo-maintainer hat and
push patches to your subtree branch. When it is time for integration,
checkout master and merge all the branches in.

Simple as hell. And faster then you ever imagine.

Additionally you can have working branches that, lets say
did-of-the-day, where you save work to 3 topics /src/bar
/src/lib4bar /src/lib4foobar. Once you are happy. You can
push them through the project branches, and delete the
working branch, or you can merge that directly into master
and when you pull master into the topic branches after a big
merge, they will have it automatically.

This is all very similar to Linux. With the subject maintainers
and all, just with fewer people.

PS: Don't forget to have a very first patch with nothing but
a common root file like GPLv2 or README. The first patch is
hard to work with.

Just my $0.017
Boaz