What is in git.git

What is in git.git

[Junio C Hamano]

The "bind commit" experiments for subproject support is coming
along rather nicely.  Near the tip of the "pu" branch, there
are:

 - read-tree --prefix;
 - write-tree --prefix and write-tree --exclude;
 - commit-tree --bind;
 - fsck-objects and convert-objects that understand "bind" lines
   in commit objects;
 - rev-list --objects that understand "bind" lines
   in commit objects;

I think the first four are more-or-less well debugged.

I am reasonably confident that I did not break rev-list for
repositories without "bind" commits, but I have no clue how
correct it is when dealing with commits with "bind" lines.  This
is the last major remaining piece of the puzzle, and the rest is
just the matter of scripting.  I'd be sending out a request for
help on the rev-list in a separate message.

There still is no barebone Porcelainish work done using these
changes.  The attached script demonstrates a superproject that
binds two subprojects with their own development histories.

-- >8 --
#!/bin/sh
rm -fr .git
git init-db
: >main
for i in 1 2
do
	echo "Version #$i of primary subproject" >main
	git add main
	git commit -a -m "`cat main`"
	git tag main-$i
done
git branch main
rm -f .git/refs/heads/master .git/index

mkdir dir
: >sub1
: >dir/sub2
for i in A B C
do
	echo "subproject sub1 ($i)" >sub1
	echo "subproject dir/sub2 ($i)" >dir/sub2
	git add sub1 dir/sub2
	git commit -a -m "subproject #$i"
	git tag subpro-$i
done
git branch subpro
rm -fr dir sub1 main
rm -f .git/refs/heads/master .git/index

git read-tree main-1
git read-tree --prefix=sub/ subpro-A
toptree=`git write-tree`
topcommit=$(echo "Initial top commit" |
git commit-tree $toptree \
	--bind `git rev-parse --verify main-1` / \
	--bind  `git rev-parse --verify subpro-A` sub/)
echo $topcommit >.git/refs/heads/master
git tag top-1A

git read-tree main-1
git read-tree --prefix=sub/ subpro-B
toptree=`git write-tree`
topcommit=$(echo "Second top commit" |
git commit-tree $toptree \
	-p `git rev-parse --verify top-1A` \
	--bind `git rev-parse --verify main-1` / \
	--bind  `git rev-parse --verify subpro-B` sub/)
echo $topcommit >.git/refs/heads/master
git tag top-1B

git read-tree main-2
git read-tree --prefix=sub/ subpro-C
toptree=`git write-tree`
topcommit=$(echo "Third top commit" |
git commit-tree $toptree \
	-p `git rev-parse --verify top-1B` \
	--bind `git rev-parse --verify main-2` / \
	--bind  `git rev-parse --verify subpro-C` sub/)
echo $topcommit >.git/refs/heads/master
git tag top-2C

git log

git rev-list --objects top-1A..top-2C |
git name-rev --tags --stdin

Re: What is in git.git

[Alexander Litvinov]

> 1. Can I bind some branch instead of tag (commit) ?
> 2. Is it possible to commit changes of subpro's file in master branch into
> subpro branch to make this changes visible to master-2 ?

One more comment: it seems to me it is not possible to make two branches on 
separate subprojects with the same name.

Re: What is in git.git

[Alexander Litvinov]

Somehow first mail was not reached the list, resending it.

> The "bind commit" experiments for subproject support is coming
> along rather nicely.  Near the tip of the "pu" branch, there
> are:
...
>
> I think the first four are more-or-less well debugged.
>
> I am reasonably confident that I did not break rev-list for
> repositories without "bind" commits, but I have no clue how
> correct it is when dealing with commits with "bind" lines.  This
> is the last major remaining piece of the puzzle, and the rest is
> just the matter of scripting.  I'd be sending out a request for
> help on the rev-list in a separate message.
>
> There still is no barebone Porcelainish work done using these
> changes.  The attached script demonstrates a superproject that
> binds two subprojects with their own development histories.

I tested this with your script. It works well. But I have found some 
downsides.

subpro and main are separate projects and master is the join of them. If I 
want to modify subpro I have to checkout subpro branch, edit files. When I 
have to got to master and bind new version of subpro to it. Worse, if I will 
edit subpro's files bined to master branch changes will go to master branch 
instead of subpro's history. As a result all other project (imagine master-2) 
that use subpro will lose this change.

1. Can I bind some branch instead of tag (commit) ?
2. Is it possible to commit changes of subpro's file in master branch into 
subpro branch to make this changes visible to master-2 ?

Thanks for attention.
Alexander Litvinov.

Re: What is in git.git

[Junio C Hamano]

I suspect there is an misunderstanding or two, because I did not
repeat what the other parts (Porcelain-ish) would do and
everything fits together.  For that, you need to refer to a
couple of earlier messages from me on the "RFC: Subprojects"
thread [*1*], and perhas use some imagination, because the
outline I did back then was done without having much of what are
in "pu" today and I may have got some details wrong.  The
message you are responding to was to give a demonstation of the
current status of how the core part works.

Alexander Litvinov <lan@ac-sw.com> writes:

>> 1. Can I bind some branch instead of tag (commit) ?

The "bind" line describes relationship among specific points in
development histories of subprojects and superprojects so it
needs to use commit object names.

If you mean by "binding a branch", to record how each subproject
relates to the toplevel project (i.e. "the subproject bound to
X/ subdirectory of the toplevel project comes from branch Y"),
that information needs to be somewhere, but recording it in the
commit object goes against the whole git philosophy.

Branch naming is a local matter.  "master" in my repository
represents a different development history of the project from
what your "master" does.  The subproject I happen to use the
"subpro" branch to keep track of might be called "sub" in yours.
Recording the name "subpro" on a "bind" line in a commit object
makes that commit object useless when you fetch such a commit
from my repository.

>> 2. Is it possible to commit changes of subpro's file in
>> master branch into subpro branch to make this changes visible
>> to master-2 ?

You are way ahead of what I am putting in "pu".  That is all
responsibility of the scripting part that use the core part;
refer to the outline in earlier messages on the subprojects
thread.

I think the above relates to this part of what you said:

> subpro and main are separate projects and master is the join
> of them. If I want to modify subpro I have to checkout subpro
> branch, edit files. When I have to got to master and bind new
> version of subpro to it.

I do not see any problem with this.  The core level tools in
"pu" supports that mode of operation.  They also support another
mode of operation, checking out the whole thing and making a
commit for subprojects.

> Worse, if I will edit subpro's files bined to master branch
> changes will go to master branch instead of subpro's history.

Simply untrue.

Even though I will use present tense (e.g. "the checkout command
notices") in the following, you should remember that the current
set of barebone Porcelainish scripts have not been taught about
any of this.

You need to keep a file that describes how your repository is
tracking the development histories of each subproject in
$GIT_DIR/bind, that would look like:

	master main=/ subpro=sub/

meaning:

	In this repository, the project whose history is kept
	track of by "master" branch binds the projects whose
	history is kept track of by "subpro" branch at sub/ and
	another project that holds the rest whose history is
	kept track by "main" branch.

You can have more than one branches for subprojects and the
combined project.  Just have more lines in $GIT_DIR/bind,
like so, to achieve that [*2*]:

     master main=/ subpro=sub/
     master20 main2=/ subpro=sub/
     master02 main2=/ subpro2=sub/
	...

To work on the example project, there are two alternative
workflows that can be supported:

1. You can be on "master" branch (i.e. checking out everything
   at the right place), and make necessary changes, be they at
   toplevel or sub/ directory.  

   The checkout command notices you are on "master" branch, and
   you are checking out a bound commit which has "bind" lines
   for / and sub/ directories.  When committing, a single "git
   commit" may make potentially three commits.  (1) If you want
   to commit changes you made to sub/ part, that is committed to
   "subpro" branch; (2) If you want to commit changes to the
   rest, that is committed to "main" branch; (3) and a commit to
   the "master" branch is made to record the state of the whole
   thing.

2. You can work on an individual subproject without bothering
   the combined project.  You can have another repository to do
   developments of the project this repository uses "subpro"
   branch to keep track of.  When that work in the other
   repository on a subproject is ready to be integrated into the
   whole, you would fetch/merge the subproject part into this
   repository from there, advancing "subpro" branch head in this
   repository.

   When this happens, you can notice that the commit on the
   "bind" line for sub/ part does not match the branch head that
   keeps track of that part of the tree (i.e. "subpro")
   anymore, and can update sub/ part by merging.

> One more comment: it seems to me it is not possible to make
> two branches on separate subprojects with the same name.

This is precisely why I said "the branch naming is a local
issue", and the commit object does not record branch name.  In
the above workflow #2, there is no reason for the other
repository that develops the sub/ subproject part to name its
primary branch "subpro".  Most likely it is named "master"
there, and the combined project would fetch its advancement by
issuing:

	$ git fetch ../subprorepo master:subpro

If you have more than one branches in the other repository and
would want to use that instead, you would fetch from that branch
not from master there.

You can also keep more than one branch for a subproject inside
the combined repository and have more than one $GIT_DIR/bind
lines that describe different superprojects that bind different
branches of the same subproject at the same location in the
corresponding superproject branch..


[Footnote]

*1* Here are a couple of key messages in the thread, that
attempt to describe how the things would fit together:

    http://article.gmane.org/gmane.comp.version-control.git/14781
    http://article.gmane.org/gmane.comp.version-control.git/14809

*2* Theoretically you could have NxMxOx... master project
branches for subprojects with N, M, O,... branches, but in
practice, the combined project is an integration field, and most
of the combinations are not something you are interested in.

Not all of the N branches in a subproject need to be directly
integrated into the whole --- most of them are used only while
coming up with the version of the subproject that is suitable
for the integration.

Re: What is in git.git

[Junio C Hamano]

Junio C Hamano <junkio@cox.net> writes:

> Alexander Litvinov <lan@ac-sw.com> writes:
>...
>> subpro and main are separate projects and master is the join
>> of them. If I want to modify subpro I have to checkout subpro
>> branch, edit files. When I have to got to master and bind new
>> version of subpro to it.
>
> I do not see any problem with this....
>...
>> Worse, if I will edit subpro's files bined to master branch
>> changes will go to master branch instead of subpro's history.
>
> Simply untrue.

Sorry, these came out somewhat in a wrong way, so let me
clarify.

What I meant was that there isn't anything coded so far that
makes your worries real issues yet, and I do not intend to code
Porcelainish scripts that are broken in the ways you see as
problems in your message.

The point you raised are valid concerns.  You need to keep them
in mind when you start writing subproject aware version of
git-checkout, git-commit and git-merge commands (among other
things I might have forgotten, but I think these three covers
pretty much everything).  You are welcome to beat me to it,
since I am not planning to do them right away.

Re: What is in git.git

[Josef Weidendorfer]

On Saturday 21 January 2006 20:37, you wrote:
> Alexander Litvinov <lan@ac-sw.com> writes:
> >> 1. Can I bind some branch instead of tag (commit) ?
> ... 
> If you mean by "binding a branch", to record how each subproject
> relates to the toplevel project (i.e. "the subproject bound to
> X/ subdirectory of the toplevel project comes from branch Y"),
> that information needs to be somewhere, but recording it in the
> commit object goes against the whole git philosophy.

The original gitlink proposal did exactly this: it recorded
the place where a subproject is bound by putting a gitlink into
a tree. This way, the binding point can be changed, and is subject to
versioning itself.

I just realized that this is not currently possible with the bind lines.
What about the following usage szenario:
- in a superproject, I use a subproject X implementing some lib by 
  binding it at X/. My Makefile recurses into X/ for this.
  This is recorded at commit point (A)
- later on, I realize I need another lib from a probject Y; I want
  to put the libs X and Y into subdirectory lib/ of my superproject;
  i.e. I bind Y at lib/Y/ and move the binding point of X to lib/X/.
  The Makefile is changed accordingly to build the subprojects.
  This is recorded at commit point (B)

A $GITDIR/bind alone will no work, as moving back to (A) would keep
the binding point of subproject, and make is broken.

I understand that "moving binding point of X from X/ to lib/X/" is not
representable within the index as a simple change. Is this the main issue
for your "against the whole git philosophy"?

I think it still is quite useful to put the binding point into bind lines of the
commit. Of course, moving a binding point has to go together with a new commit.

> You need to keep a file that describes how your repository is
> tracking the development histories of each subproject in
> $GIT_DIR/bind, that would look like:
> 
> 	master main=/ subpro=sub/

What about putting $GITDIR/bind information directly into reference files?

 $HOME/gitproj> cat .git/refs/heads/master
 92347432598...
 bind main=/
 bind subpro=sub/

This way, you can rename/copy heads, and the binding info will stay with
the commit.
This also is nice for subprojects that do not need the bind lines, but similar
to current cogito subproject proposal:

  92347432598...
  main=/
  subpro=sub/

would be the same superproject/subproject configuration without commiting
bind lines.

Josef

Re: What is in git.git

[Junio C Hamano]

Josef Weidendorfer <Josef.Weidendorfer@gmx.de> writes:

> The original gitlink proposal did exactly this: it recorded
> the place where a subproject is bound by putting a gitlink into
> a tree. This way, the binding point can be changed, and is subject to
> versioning itself.
>
> I just realized that this is not currently possible with the bind lines.
> What about the following usage szenario:
> - in a superproject, I use a subproject X implementing some lib by 
>   binding it at X/. My Makefile recurses into X/ for this.
>   This is recorded at commit point (A)
> - later on, I realize I need another lib from a probject Y; I want
>   to put the libs X and Y into subdirectory lib/ of my superproject;
>   i.e. I bind Y at lib/Y/ and move the binding point of X to lib/X/.
>   The Makefile is changed accordingly to build the subprojects.
>   This is recorded at commit point (B)

The original gitlink proposal records commit object name in the
link object itself, so do bind lines in the commit object in
bound commit proposal.  In either way, you need to deal with the
subproject relocation at the Porcelain level.

I was hoping that, upon seeing these two commits (let's say we
are dealing with two-way merge aka "checkout"):

	In commit 1:
		bind xxxxx... X/

	In commit 2:
		bind yyyyy... lib/X/
                bind zzzzz... lib/Y/

the tool could notice that xxxxx... and yyyyy... are related in
their ancestry chain, detect the relocation of subprojects, and
update the $GIT_DIR/bind file (maybe with some help from the end
user).  We can do something similar in gitlink approach as well.

> A $GITDIR/bind alone will no work, as moving back to (A) would keep
> the binding point of subproject, and make is broken.

I do not see why.  $GIT_DIR/bind can be adjusted by the tool
upon checkout to reflect the reorganized tree.

> What about putting $GITDIR/bind information directly into reference files?
>
>  $HOME/gitproj> cat .git/refs/heads/master
>  92347432598...
>  bind main=/
>  bind subpro=sub/

I think that would also work.  Although I do not immediately see
major difference in expressiveness either way, that may be a
cleaner way to achieve what we want to do.

Re: What is in git.git

[Josef Weidendorfer]

On Sunday 22 January 2006 03:44, you wrote:
> 	In commit 1:
> 		bind xxxxx... X/

Ah, OK, sorry.
This was a misunderstanding from my side; please forget the
other suggestion about change reference files, too ;-)

Josef

Re: What is in git.git

[Petr Baudis]

Just-in-case panic response...

Dear diary, on Sun, Jan 22, 2006 at 12:33:25AM CET, I got a letter
where Josef Weidendorfer <Josef.Weidendorfer@gmx.de> said that...
> > You need to keep a file that describes how your repository is
> > tracking the development histories of each subproject in
> > $GIT_DIR/bind, that would look like:
> > 
> > 	master main=/ subpro=sub/
> 
> What about putting $GITDIR/bind information directly into reference files?
> 
>  $HOME/gitproj> cat .git/refs/heads/master
>  92347432598...
>  bind main=/
>  bind subpro=sub/
> 
> This way, you can rename/copy heads, and the binding info will stay with
> the commit.

Please don't. I can see no real advantage to separate files (except for
the very rare case of rename/copy/removal of heads) except for massive
porcelain breakage and significant clutter-up of all the code dealing
with refs.

Please leave our poor refs alone. :) They are fine as they are - simple
one-value thing.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Of the 3 great composers Mozart tells us what it's like to be human,
Beethoven tells us what it's like to be Beethoven and Bach tells us
what it's like to be the universe.  -- Douglas Adams

Re: What is in git.git

[Daniel Barkalow]

On Sun, 22 Jan 2006, Josef Weidendorfer wrote:

> On Saturday 21 January 2006 20:37, you wrote:
> > Alexander Litvinov <lan@ac-sw.com> writes:
> > >> 1. Can I bind some branch instead of tag (commit) ?
> > ... 
> > If you mean by "binding a branch", to record how each subproject
> > relates to the toplevel project (i.e. "the subproject bound to
> > X/ subdirectory of the toplevel project comes from branch Y"),
> > that information needs to be somewhere, but recording it in the
> > commit object goes against the whole git philosophy.
> 
> The original gitlink proposal did exactly this: it recorded
> the place where a subproject is bound by putting a gitlink into
> a tree. This way, the binding point can be changed, and is subject to
> versioning itself.
> 
> I just realized that this is not currently possible with the bind lines.
> What about the following usage szenario:
> - in a superproject, I use a subproject X implementing some lib by 
>   binding it at X/. My Makefile recurses into X/ for this.
>   This is recorded at commit point (A)
> - later on, I realize I need another lib from a probject Y; I want
>   to put the libs X and Y into subdirectory lib/ of my superproject;
>   i.e. I bind Y at lib/Y/ and move the binding point of X to lib/X/.
>   The Makefile is changed accordingly to build the subprojects.
>   This is recorded at commit point (B)
> 
> A $GITDIR/bind alone will no work, as moving back to (A) would keep
> the binding point of subproject, and make is broken.

I think you're misunderstanding the use of the "bind" file or equivalent. 
It isn't a configuration file that specifies where the subprojects go; 
it's a state file that tracks where the subprojects currently are. The 
commits by themselves are sufficient to indentify the locations of the 
subprojects, and the bind file would be written by "git checkout" reading 
a commit with subprojects. It's used to create the next commit, in much 
the same way that MERGE_HEAD is used to create the next commit by storing 
information between the git call that starts a merge that needs user 
interaction and the git call to commit the merge.

So moving back to (A) wouldn't keep the binding point of subproject, 
because it would rewrite bind to what it had been.

I'm going to suggest again keeping this information in the index file (but 
not in the index data structure, so the changes to the code are only in 
the library routines to read and write the file, and, of course, anything 
that's actually trying to manipulate the binding locations). I started 
working on a patch to pu to skip S_IFDIR entries from the index file when 
building the table in memory, and that was straightforward, but I got into 
sysadmin issues when I was going to test giving it something to skip.

	-Daniel
*This .sig left intentionally blank*

Re: What is in git.git

[Junio C Hamano]

Daniel Barkalow <barkalow@iabervon.org> writes:

> I think you're misunderstanding the use of the "bind" file or equivalent. 
>...
> So moving back to (A) wouldn't keep the binding point of subproject, 
> because it would rewrite bind to what it had been.

A lot better said than my version of the response.  Thanks.

> I'm going to suggest again keeping this information in the index file (but 
> not in the index data structure, so the changes to the code are only in 
> the library routines to read and write the file, and, of course, anything 
> that's actually trying to manipulate the binding locations). I started 
> working on a patch to pu to skip S_IFDIR entries from the index file when 
> building the table in memory, and that was straightforward, but I got into 
> sysadmin issues when I was going to test giving it something to skip.

I have been thinking about this one, and having read that
read-cache code I think the coding is not too involved.

My current inclination is to use the same version number (2) by
default and promote it to a new version number (3) once you add
subproject-binding information to the index file.  Then current
tools would keep working on repositories created or operated
upon with the new tools, as long as the project does not use the
new feature.

Re: What is in git.git

[Daniel Barkalow]

On Sun, 22 Jan 2006, Junio C Hamano wrote:

> I have been thinking about this one, and having read that
> read-cache code I think the coding is not too involved.
> 
> My current inclination is to use the same version number (2) by
> default and promote it to a new version number (3) once you add
> subproject-binding information to the index file.  Then current
> tools would keep working on repositories created or operated
> upon with the new tools, as long as the project does not use the
> new feature.

I think bumping the index file version number shouldn't be too big a deal; 
index files are rarely used by different versions, except for the case 
where you've installed a new version of git, and that's generally a later 
version, not an earlier one.

It might make sense to add logic to make the stable series accept version 
3 files without special entries, though. And maybe future-proof by having 
it look for unexpected flags and give a "too new version" error for them, 
even if the version number is within range. (I.e., if we add more special 
entry types later, it would tell you if your index file uses a feature 
that the version of git you're using doesn't support, even if the version 
number hasn't changed, and we reserve changing the version number for 
things where the file wouldn't read right at all or something previously 
legal changes meaning.)

On a side topic, is there any reason not to convert ls-tree to use struct
tree, and kill the other tree-object-parsing code, which doesn't seem to 
be used by anything else?

	-Daniel
*This .sig left intentionally blank*

Re: What is in git.git

[Junio C Hamano]

Daniel Barkalow <barkalow@iabervon.org> writes:

> On a side topic, is there any reason not to convert ls-tree to use struct
> tree, and kill the other tree-object-parsing code, which doesn't seem to 
> be used by anything else?

ls-tree _might_ be doing something struct tree interface does
not support, but I do not know.

If you are inclined to, please go wild ;-).

Re: What is in git.git

[Junio C Hamano]

Josef Weidendorfer <Josef.Weidendorfer@gmx.de> writes:

> I understand that "moving binding point of X from X/ to lib/X/" is not
> representable within the index as a simple change. Is this the main issue
> for your "against the whole git philosophy"?

No.  I meant an exposure of local branch names by recording them
in the commit object, and nothing else, by that comment.

Instead of using an extra $GIT_DIR/bind file extending what we
record in the index file is OK.  $GIT_INDEX_FILE, $GIT_DIR/HEAD,
and $GIT_DIR/bind pretty much go hand-in-hand anyway and they
_are_ local to the repository, just like branch names are, so I
do not have any problem with using local branch names in these
places, but not in a commit object (or gitlink object for that
matter).