Re: [gambit-list] Separating generated files?

[Christian Jaeger <christian@pflanze.mine.nu>]

I wrote:
> Michael J Gruber wrote:
>> I wonder whether a clever use of "excludes" and GIT_DIR would allow
>> tracking the different filesets in the same dir, but using different
>> repos. I'm just afraid it's a fragile setup, in the sense that it relies
>> on config stuff which is not tracked (and thus not reproduced
>> automatically on clone).
>
> I expect that using a superproject repository to tie together the two 
> repositories is good and necessary because it is the link that allows 
> to specify which commit in the repo of generated files belongs 
> together with a commit in the repo of source files. So just using two 
> separate repositories without making them submodules of a superproject 
> does not seem to be a good idea to me.

(In the meantime I've read the following pages:
http://nopugs.com/2008/09/06/ext-tutorial
http://nopugs.com/2008/09/04/why-ext
http://flavoriffic.blogspot.com/2008/05/managing-git-submodules-with-gitrake.html
(a post to the latter article suggests to use subtree merging instead, 
but that would be a very bad match for our use case; the mentioned 
problem of merging of the git superproject makes me think, though--the 
superproject could be updated only by the one person doing the publish 
onto the public repository, but then it leaves the problem of handling 
merges by developers completely unsolved.)
)

I'm starting to think that maybe a better idea than the 
superproject+2submodules approach would be just using the two 
repositories ("source" + "build"), and storing the linking information 
inside the "build" repository (by adding the source repository commitid 
to every commit message in the build repository [or using tags, but that 
doesn't seem a better idea]), and use a program that is able to check 
out the matching "build" repository for a given "source" repository 
checkout.

I'm willing to write this program (let's call it 
"intergit-find-matching-commit-in" for the purpose of this email); 
question: which language to write it in, is Perl good? (C would be a 
hassle for Windows users because of the C compiler requirement; shell 
may be too limited.)

Description of the workings in more detail:

- one would work with the "source" repository just as one would with any 
project only employing one repository; do some changes to the project, 
commit them, test them (includes regeneration of generated files);

- once in a while one would commit the current generated files in the 
"build" repository; by either (a) using a make target (like "make 
commit_generated") which runs something like

  eval "cd build; git commit -m 'generated files for source repository 
commit `git rev-parse HEAD`'"

or (b) setting up a build/.git/hooks/commit-msg script which appends 
'generated files for source repository commit `git rev-parse HEAD`' line 
to the commit message given from running "cd build; git commit -a" manually.

- for publication, one would push both the "source" as well as the 
"build" repository (i.e. "cd build; git push; cd ..; git push")

- for checkout (our "make update" make target), about the following 
would happen:

   git pull
   eval "(cd build; git checkout `intergit-find-matching-commit-in build`)"

where "intergit-find-matching-commit-in build" would first refresh an 
index of the links (iterate over all unseen commits, parse commit 
messages for /source repository commit (\w+)/ and store $1 => 
$commitid_in_build_repo mappings in the index), then go through "git log 
--pretty=format:%H" (should I also specify --topo-order (or 
--date-order)?) looking up the commitids in the index, stopping at the 
first match and outputting the mapped $commitid_in_build_repo.

This way, the "latest" or "probably best-matching" corresponding commit 
in the "build" repo can always be found, even if the "source" repo is 
ahead, which should allow building the compiler even if none is 
previously installed. This workflow seems more natural than the 
superproject+submodules approach, and it seems to entail no hassle with 
merge issues (only the "source" repo really needs proper merging; 
merging the "build" repo would only be worthwhile for maintaining the 
history, and as mentioned if there are conflicts, one would probably 
usually just regenerate the files there; there's no need to maintain 
linking info (with associated merge etc issues) in a separate entity 
(superproject) anymore, and during development, commits to the "build" 
repo need only be done if backwards-incompatible changes have been 
introduced).

Does anyone else think this is sane/interesting? Should I go ahead 
implementing this? Any comments, like on how the interface of the 
intergit-find-matching-commit-in tool should look like?

Christian.