Question about handling of heterogeneous repositories

Question about handling of heterogeneous repositories

[Alex Riesen]

Hi,

it is sometimes the case that a project consists of parts which are
unrelated to each other, and only thing in common between them is that
they all are used in that particular project. For example a program
uses some library and the developer(s) of that program would like to
have the source of that library somewhere close. Well, for this simple
example one could just use two repositories, laid close to each other
in a directory, like project/lib and project/prog.
Now, if I make the example a bit more complex and say, that the
developers of the program are the developers in that project and
change everything under project/ directory, including
project/library/. They are also good people and ready to give the
changes to the library upstream.

How do they achieve that, without sending project/ and project/program/?

For everyone who have an experience with ClearCase or Perforce (I'm
sorry for mentioning it) it is what the "mappings" are often used for:
a project is build together from different parts, which can be worked
on separately.

I'm trying to introduce git at work, but have to prepare myself for
possible questions first, and this is one of them :)

Greetings,
Alex Riesen

Re: Question about handling of heterogeneous repositories

[Andreas Ericsson]

Alex Riesen wrote:
> Hi,
> 
> it is sometimes the case that a project consists of parts which are
> unrelated to each other, and only thing in common between them is that
> they all are used in that particular project. For example a program
> uses some library and the developer(s) of that program would like to
> have the source of that library somewhere close. Well, for this simple
> example one could just use two repositories, laid close to each other
> in a directory, like project/lib and project/prog.
> Now, if I make the example a bit more complex and say, that the
> developers of the program are the developers in that project and
> change everything under project/ directory, including
> project/library/. They are also good people and ready to give the
> changes to the library upstream.
> 
> How do they achieve that, without sending project/ and project/program/?
> 
> For everyone who have an experience with ClearCase or Perforce (I'm
> sorry for mentioning it) it is what the "mappings" are often used for:
> a project is build together from different parts, which can be worked
> on separately.
> 
> I'm trying to introduce git at work, but have to prepare myself for
> possible questions first, and this is one of them :)
> 

We do like this;

core
core/gui
core/lib

$ cat .gitignore
gui
lib

This is also nice because it lets the gui maintainers have the gui as 
the root with the core and lib parts as subdirectories. Everyone has 
their own responsibility checked out at top-level with other pieces 
below it. It's easy enough to script a pull of all repos so everyone's 
up to sync and everybody's happy.

It would certainly be nicer to have git ignore directories that have the 
".git" directory (so long as it's not the top of the repo, that is), but 
I haven't had the energy to fix that when there's already a solution 
that's simple enough and quite adequate.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: Question about handling of heterogeneous repositories

[Alex Riesen]

Andreas Ericsson, Tue, Nov 22, 2005 20:40:50 +0100:
> >it is sometimes the case that a project consists of parts which are
> >unrelated to each other, and only thing in common between them is that
> >they all are used in that particular project. For example a program
> >uses some library and the developer(s) of that program would like to
> >have the source of that library somewhere close. Well, for this simple
> >example one could just use two repositories, laid close to each other
> >in a directory, like project/lib and project/prog.
> >Now, if I make the example a bit more complex and say, that the
> >developers of the program are the developers in that project and
> >change everything under project/ directory, including
> >project/library/. They are also good people and ready to give the
> >changes to the library upstream.
> >
> >How do they achieve that, without sending project/ and project/program/?
> >
> >For everyone who have an experience with ClearCase or Perforce (I'm
> >sorry for mentioning it) it is what the "mappings" are often used for:
> >a project is build together from different parts, which can be worked
> >on separately.
> >
> >I'm trying to introduce git at work, but have to prepare myself for
> >possible questions first, and this is one of them :)
> >
> 
> We do like this;
> 
> core
> core/gui
> core/lib
> 
> $ cat .gitignore
> gui
> lib
> 
> This is also nice because it lets the gui maintainers have the gui as 
> the root with the core and lib parts as subdirectories. Everyone has 
> their own responsibility checked out at top-level with other pieces 
> below it. It's easy enough to script a pull of all repos so everyone's 
> up to sync and everybody's happy.

And than, do you have to announce a special procedure everyone's is to
execute after a clone so all subrepos are cloned? How do you handle
common configs and clone options in subrepos? It also would be nice to
have branches visible not only in in subrepos but in top-repo as well,
and the other way around.

I'm also wondering, what implications core/lib/.git/{config,remotes/,refs/}
under control in core/ can have?..

> It would certainly be nicer to have git ignore directories that have the 
> ".git" directory (so long as it's not the top of the repo, that is), but 
> I haven't had the energy to fix that when there's already a solution 
> that's simple enough and quite adequate.

BTW, will something like "*/.git/*" in info/exclude work? IOW, does *
match a "/"?

Re: Question about handling of heterogeneous repositories

[Petr Baudis]

Dear diary, on Wed, Nov 23, 2005 at 12:22:28AM CET, I got a letter
where Alex Riesen <raa.lkml@gmail.com> said that...
> Andreas Ericsson, Tue, Nov 22, 2005 20:40:50 +0100:
> > >it is sometimes the case that a project consists of parts which are
> > >unrelated to each other, and only thing in common between them is that
> > >they all are used in that particular project. For example a program
> > >uses some library and the developer(s) of that program would like to
> > >have the source of that library somewhere close. Well, for this simple
> > >example one could just use two repositories, laid close to each other
> > >in a directory, like project/lib and project/prog.
> > >Now, if I make the example a bit more complex and say, that the
> > >developers of the program are the developers in that project and
> > >change everything under project/ directory, including
> > >project/library/. They are also good people and ready to give the
> > >changes to the library upstream.
> > >
> > >How do they achieve that, without sending project/ and project/program/?
> > >
> > >For everyone who have an experience with ClearCase or Perforce (I'm
> > >sorry for mentioning it) it is what the "mappings" are often used for:
> > >a project is build together from different parts, which can be worked
> > >on separately.
> > >
> > >I'm trying to introduce git at work, but have to prepare myself for
> > >possible questions first, and this is one of them :)

This is something e.g. Cogito wants to support, but does not yet.
Patches welcome.

> > It would certainly be nicer to have git ignore directories that have the 
> > ".git" directory (so long as it's not the top of the repo, that is), but 
> > I haven't had the energy to fix that when there's already a solution 
> > that's simple enough and quite adequate.
> 
> BTW, will something like "*/.git/*" in info/exclude work? IOW, does *
> match a "/"?

Nope, but try just '.git' - in case it is not a pathname but just a
filename (or dirname, for that matter), it will recursively apply to all
the subtrees.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.

Re: Question about handling of heterogeneous repositories

[Alex Riesen]

Petr Baudis, Sun, Nov 27, 2005 14:11:47 +0100:
> > > >For everyone who have an experience with ClearCase or Perforce (I'm
> > > >sorry for mentioning it) it is what the "mappings" are often used for:
> > > >a project is build together from different parts, which can be worked
> > > >on separately.
> > > >
> > > >I'm trying to introduce git at work, but have to prepare myself for
> > > >possible questions first, and this is one of them :)
> 
> This is something e.g. Cogito wants to support, but does not yet.
> Patches welcome.

I wouldn't know what to patch, having no clear picture of the approach
myself, and especially when I don't feel safe using the solution. For
example, how do you go about moving/renaming files between subrepos?
Rename detection will not work, which will be unexpected...

BTW, how does git-mv behave for out-of-tree renaming? How about
inter-repo renaming (remove+add)?

> > > It would certainly be nicer to have git ignore directories that have the 
> > > ".git" directory (so long as it's not the top of the repo, that is), but 
> > > I haven't had the energy to fix that when there's already a solution 
> > > that's simple enough and quite adequate.
> > 
> > BTW, will something like "*/.git/*" in info/exclude work? IOW, does *
> > match a "/"?
> 
> Nope, but try just '.git' - in case it is not a pathname but just a
> filename (or dirname, for that matter), it will recursively apply to all
> the subtrees.

well, it ignored the ".git"s in the subdirs, not _the_ subdirectories.
I think that can be helped by putting the directories themselves into
.gitignore lists.

Re: Question about handling of heterogeneous repositories

[Alex Riesen]

On 11/30/05, Josef Weidendorfer <Josef.Weidendorfer@gmx.de> wrote:
> > BTW, how does git-mv behave for out-of-tree renaming? How about
> > inter-repo renaming (remove+add)?
>
> git-mv will fail. The source has to appear in git-ls-files, and the
> target simply is given to "git-update-index -add", which will return an
> 'outside repository' error. Hmmm... git-mv should check this itself.
>
> To detect inter-repository renaming, one has to see both repositories.
> Currently, git/cogito etc. commands only work with one repository only.

This case is somewhat special: one is _guaranteed_ to see both repositories.

PS: Im'restoring git@vger.kernel.org in cc-list, the discussion could be useful

Re: Question about handling of heterogeneous repositories

[Johannes Schindelin]

Hi,

On Wed, 30 Nov 2005, Alex Riesen wrote:

> On 11/30/05, Josef Weidendorfer <Josef.Weidendorfer@gmx.de> wrote:
> >
> > To detect inter-repository renaming, one has to see both repositories. 
> > Currently, git/cogito etc. commands only work with one repository 
> > only.
> 
> This case is somewhat special: one is _guaranteed_ to see both 
> repositories.

It is even more so: it is guaranteed that git cannot reconstruct this 
as a move operation.

Hth,
Dscho

Re: Question about handling of heterogeneous repositories

[Josef Weidendorfer]

On Wednesday 30 November 2005 14:58, Johannes Schindelin wrote:
> Hi,
> 
> On Wed, 30 Nov 2005, Alex Riesen wrote:
> 
> > On 11/30/05, Josef Weidendorfer <Josef.Weidendorfer@gmx.de> wrote:
> > >
> > > To detect inter-repository renaming, one has to see both repositories. 
> > > Currently, git/cogito etc. commands only work with one repository 
> > > only.
> > 
> > This case is somewhat special: one is _guaranteed_ to see both 
> > repositories.
> 
> It is even more so: it is guaranteed that git cannot reconstruct this 
> as a move operation.

Say it in another way: because currently, git commands only work in the
scope of one repository, there is no need to want to detect an
inter-repository rename.

Git has no explicit support for renaming, but can do it. Why should it
be a problem to write a tool which is able to detect inter-repository
renames? Still, such a search probably would be very expensive.

git-mv could support this. Additional to deleting from
one repository and adding to the new, it could generate a remark for
the commit message, like:

 Added file xxx is from remote repo '/path/myrepo' with name 'yyy'
 (file SHA1: 123445... from remote commit SHA1: 345667...)

This info could be interesting both for humans and a history browsing tool.
The tool still would have to know where to look for repo '/path/myrepo'.
It could also work for moving subdirectories (ie. trees instead of files).

Josef

Re: Question about handling of heterogeneous repositories

[Alex Riesen]

On 11/30/05, Josef Weidendorfer <Josef.Weidendorfer@gmx.de> wrote:
> git-mv could support this. Additional to deleting from
> one repository and adding to the new, it could generate a remark for
> the commit message, like:
>
>  Added file xxx is from remote repo '/path/myrepo' with name 'yyy'
>  (file SHA1: 123445... from remote commit SHA1: 345667...)

That's sounds very tempting to special-case for nested repos :)

Maybe use .git/objects/info/alternates in top-level repo? It may
have side-effects of the new objects ending up in top-level repo
if commited from top-level directory. Even then, the tools can
be tought to stop looking for GIT_DIR in the directory closest
to the file being commited:
  $ pwd
  .../top
  $ git-commit sub1/dir/file
  ... uses repo in .../top/sub1 instead of $(pwd)/../top/.git

Looks very messy, with a potential for lots of corner cases and
not that tempting anymore :(

> This info could be interesting both for humans and a history browsing tool.
> The tool still would have to know where to look for repo '/path/myrepo'.
> It could also work for moving subdirectories (ie. trees instead of files).

Well, these are just lots of files. I can't really think of case were
I'd need to know about moving a directory without knowing what
happened to the files.

>
> Josef
>

Re: Question about handling of heterogeneous repositories

[Josef Weidendorfer]

On Wednesday 30 November 2005 16:01, Alex Riesen wrote:
> On 11/30/05, Josef Weidendorfer <Josef.Weidendorfer@gmx.de> wrote:
> > git-mv could support this. Additional to deleting from
> > one repository and adding to the new, it could generate a remark for
> > the commit message, like:
> >
> >  Added file xxx is from remote repo '/path/myrepo' with name 'yyy'
> >  (file SHA1: 123445... from remote commit SHA1: 345667...)
> 
> That's sounds very tempting to special-case for nested repos :)

If the repo path is an URL, this works quite general. I tried
to avoid adding local head names to this remark...

> Maybe use .git/objects/info/alternates in top-level repo?

Or better the .git/remotes namespace...
Still, this is misusing repo-local config for multiple repos.
Seems quite confusing for me.

> It may 
> have side-effects of the new objects ending up in top-level repo
> if commited from top-level directory. Even then, the tools can
> be tought to stop looking for GIT_DIR in the directory closest
> to the file being commited:
>   $ pwd
>   .../top
>   $ git-commit sub1/dir/file
>   ... uses repo in .../top/sub1 instead of $(pwd)/../top/.git
> 
> Looks very messy, with a potential for lots of corner cases and
> not that tempting anymore :(

git-mv would have to do exactly something like this to detect
inter-repo renames at all ;-)
But in general, this is really messy.

See my suggestion in a previous thread with similar topic
 http://www.gelato.unsw.edu.au/archives/git/0511/12028.html

Josef

Re: Question about handling of heterogeneous repositories

[Alex Riesen]

On 11/30/05, Josef Weidendorfer <Josef.Weidendorfer@gmx.de> wrote:
> git-mv would have to do exactly something like this to detect
> inter-repo renames at all ;-)
> But in general, this is really messy.

That's is what I'm most afraid of.

> See my suggestion in a previous thread with similar topic
>  http://www.gelato.unsw.edu.au/archives/git/0511/12028.html

Oh, thanks! Pity, I missed that discussion,
Pasky's subprojects is exactly what I am looking for.