[RFC PATCH] Support gitlinks in fast-import/export.

[RFC PATCH] Support gitlinks in fast-import/export.

[Alexander Gavrilov]

Currently fast-import/export cannot be used for
repositories with submodules. This patch extends
the relevant programs to make them correctly
process gitlinks.

Signed-off-by: Alexander Gavrilov <angavrilov@gmail.com>
---

	I noticed that fast-export & fast-import cannot work with
	repositories using submodules: import complains about
	an invalid mode, and export fails while trying to open the SHA
	as a blob.

	As I didn't see any particular reason for it to be so, I tried to
	implement support for gitlinks.

	What I'm unsure of is, should fast-export try to reuse commit
	marks for gitlinks where it happened to recognize the object,
	or always output the SHA as it is stored in the tree?

	-- Alexander

 Documentation/git-fast-import.txt |    3 +++
 builtin-fast-export.c             |   18 +++++++++++++++---
 fast-import.c                     |   12 +++++++++++-
 3 files changed, 29 insertions(+), 4 deletions(-)

diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt
index 395c055..80c591a 100644
--- a/Documentation/git-fast-import.txt
+++ b/Documentation/git-fast-import.txt
@@ -481,6 +481,9 @@ in octal.  Git only supports the following modes:
   what you want.
 * `100755` or `755`: A normal, but executable, file.
 * `120000`: A symlink, the content of the file will be the link target.
+* `160000`: A gitlink, SHA-1 of the object refers to a commit in
+  another repository. Git links can only be specified by SHA or through
+  a commit mark. They are used to implements submodules.
 
 In both formats `<path>` is the complete path of the file to be added
 (if not already existing) or modified (if already existing).
diff --git a/builtin-fast-export.c b/builtin-fast-export.c
index d0a462f..14b1549 100644
--- a/builtin-fast-export.c
+++ b/builtin-fast-export.c
@@ -123,8 +123,19 @@ static void show_filemodify(struct diff_queue_struct *q,
 			printf("D %s\n", spec->path);
 		else {
 			struct object *object = lookup_object(spec->sha1);
-			printf("M %06o :%d %s\n", spec->mode,
-			       get_object_mark(object), spec->path);
+			int mark = object ? get_object_mark(object) : 0;
+
+			if (mark)
+				printf("M %06o :%d %s\n", spec->mode,
+				       mark, spec->path);
+			else {
+				if (!S_ISGITLINK(spec->mode))
+					die("Unknown object added: %s at %s",
+					    sha1_to_hex(spec->sha1), spec->path);
+
+				printf("M %06o %s %s\n", spec->mode,
+				       sha1_to_hex(spec->sha1), spec->path);
+			}
 		}
 	}
 }
@@ -183,7 +194,8 @@ static void handle_commit(struct commit *commit, struct rev_info *rev)
 				    "", &rev->diffopt);
 
 	for (i = 0; i < diff_queued_diff.nr; i++)
-		handle_object(diff_queued_diff.queue[i]->two->sha1);
+		if (!S_ISGITLINK(diff_queued_diff.queue[i]->two->mode))
+			handle_object(diff_queued_diff.queue[i]->two->sha1);
 
 	mark_object(&commit->object);
 	if (!is_encoding_utf8(encoding))
diff --git a/fast-import.c b/fast-import.c
index e72b286..e7977c1 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -1868,6 +1868,7 @@ static void file_change_m(struct branch *b)
 	case S_IFREG | 0644:
 	case S_IFREG | 0755:
 	case S_IFLNK:
+	case S_IFGITLINK:
 	case 0644:
 	case 0755:
 		/* ok */
@@ -1900,7 +1901,16 @@ static void file_change_m(struct branch *b)
 		p = uq.buf;
 	}
 
-	if (inline_data) {
+	if (S_ISGITLINK(mode)) {
+		if (inline_data)
+			die("Git links cannot be specified 'inline': %s",
+				command_buf.buf);
+		else if (oe) {
+			if (oe->type != OBJ_COMMIT)
+				die("Not a commit (actually a %s): %s",
+					typename(oe->type), command_buf.buf);
+		}
+	} else if (inline_data) {
 		static struct strbuf buf = STRBUF_INIT;
 
 		if (p != uq.buf) {
-- 
1.5.6.3.18.gfe82

Re: [RFC PATCH] Support gitlinks in fast-import/export.

[Johannes Schindelin]

Hi,

On Fri, 18 Jul 2008, Alexander Gavrilov wrote:

> 	What I'm unsure of is, should fast-export try to reuse commit
> 	marks for gitlinks where it happened to recognize the object,
> 	or always output the SHA as it is stored in the tree?

Are they commit marks?  No.  So they should be handled as marks, just as 
those for blobs and trees.

(They are commit marks in the _submodule_, but that does not matter here.)

> diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt
> index 395c055..80c591a 100644
> --- a/Documentation/git-fast-import.txt
> +++ b/Documentation/git-fast-import.txt
> @@ -481,6 +481,9 @@ in octal.  Git only supports the following modes:
>    what you want.
>  * `100755` or `755`: A normal, but executable, file.
>  * `120000`: A symlink, the content of the file will be the link target.
> +* `160000`: A gitlink, SHA-1 of the object refers to a commit in
> +  another repository. Git links can only be specified by SHA or through
> +  a commit mark. They are used to implements submodules.

s/\(implement\)s/\1/

> diff --git a/builtin-fast-export.c b/builtin-fast-export.c
> index d0a462f..14b1549 100644
> --- a/builtin-fast-export.c
> +++ b/builtin-fast-export.c
> @@ -123,8 +123,19 @@ static void show_filemodify(struct diff_queue_struct *q,
>  			printf("D %s\n", spec->path);
>  		else {
>  			struct object *object = lookup_object(spec->sha1);
> -			printf("M %06o :%d %s\n", spec->mode,
> -			       get_object_mark(object), spec->path);
> +			int mark = object ? get_object_mark(object) : 0;

As I said, that looks wrong.  Maybe we have to fake objects for the 
gitlinks.

> @@ -183,7 +194,8 @@ static void handle_commit(struct commit *commit, struct rev_info *rev)
>  				    "", &rev->diffopt);
>  
>  	for (i = 0; i < diff_queued_diff.nr; i++)
> -		handle_object(diff_queued_diff.queue[i]->two->sha1);
> +		if (!S_ISGITLINK(diff_queued_diff.queue[i]->two->mode))
> +			handle_object(diff_queued_diff.queue[i]->two->sha1);

Why?  You do not want to export changes in the submodules?

> diff --git a/fast-import.c b/fast-import.c
> index e72b286..e7977c1 100644
> --- a/fast-import.c
> +++ b/fast-import.c

I'll let Shawn comment on that.  Oh, wait, it's his last day in work 
today.  Maybe I have something useful to say about this part of the patch, 
then, to save Shawn some work.

> @@ -1900,7 +1901,16 @@ static void file_change_m(struct branch *b)
>  		p = uq.buf;
>  	}
>  
> -	if (inline_data) {
> +	if (S_ISGITLINK(mode)) {
> +		if (inline_data)
> +			die("Git links cannot be specified 'inline': %s",
> +				command_buf.buf);
> +		else if (oe) {
> +			if (oe->type != OBJ_COMMIT)
> +				die("Not a commit (actually a %s): %s",
> +					typename(oe->type), command_buf.buf);

How is that supposed to work?  Do I understand correctly that you require 
the user to construct a commit object for the gitlink?  That would be 
actively wrong.

Oh, and your patch lacks test cases that demonstrate how you envisage the 
whole thing to work.

Ciao,
Dscho

Re: [RFC PATCH] Support gitlinks in fast-import/export.

[Alexander Gavrilov]

Hello,

On Friday 18 July 2008 21:36:26 Johannes Schindelin wrote:
> > 	What I'm unsure of is, should fast-export try to reuse commit
> > 	marks for gitlinks where it happened to recognize the object,
> > 	or always output the SHA as it is stored in the tree?
 
> Are they commit marks?  No.  So they should be handled as marks, just as 
> those for blobs and trees.
> 
> (They are commit marks in the _submodule_, but that does not matter here.)

Well, I was thinking of that unlikely case when the master module and the submodule
are in the same repository and are exported together. But probably it is best to just
output the SHA after all.

> >  	for (i = 0; i < diff_queued_diff.nr; i++)
> > -		handle_object(diff_queued_diff.queue[i]->two->sha1);
> > +		if (!S_ISGITLINK(diff_queued_diff.queue[i]->two->mode))
> > +			handle_object(diff_queued_diff.queue[i]->two->sha1);
> 
> Why?  You do not want to export changes in the submodules?

handle_object opens the sha as a blob and outputs it. As gitlinks aren't blobs, it won't work.
And if the submodule is in a separate repository, there is nothing to open anyway.

Simultaneously reading commits from the submodule repository is a whole different level
of complexity. I'm afraid I'm not up to it yet.

> > diff --git a/builtin-fast-export.c b/builtin-fast-export.c
> > index d0a462f..14b1549 100644
> > --- a/builtin-fast-export.c
> > +++ b/builtin-fast-export.c
> > @@ -123,8 +123,19 @@ static void show_filemodify(struct diff_queue_struct *q,
> >  			printf("D %s\n", spec->path);
> >  		else {
> >  			struct object *object = lookup_object(spec->sha1);
> > -			printf("M %06o :%d %s\n", spec->mode,
> > -			       get_object_mark(object), spec->path);
> > +			int mark = object ? get_object_mark(object) : 0;
> 
> As I said, that looks wrong.  Maybe we have to fake objects for the 
> gitlinks.

I tried to avoid faking blobs and stick to the interface of the M command itself.

> > @@ -1900,7 +1901,16 @@ static void file_change_m(struct branch *b)
> >  		p = uq.buf;
> >  	}
> >  
> > -	if (inline_data) {
> > +	if (S_ISGITLINK(mode)) {
> > +		if (inline_data)
> > +			die("Git links cannot be specified 'inline': %s",
> > +				command_buf.buf);
> > +		else if (oe) {
> > +			if (oe->type != OBJ_COMMIT)
> > +				die("Not a commit (actually a %s): %s",
> > +					typename(oe->type), command_buf.buf);
> 
> How is that supposed to work?  Do I understand correctly that you require 
> the user to construct a commit object for the gitlink?  That would be 
> actively wrong.

There are three forms of the M command:

M mode inline some/path
...some data...

M mode :mark some/path

M mode SHA some/path

For usual files the mark must be created by the 'blob' command,
and the SHA must refer to an existing blob. This hunk makes fast-import
require for gitlinks a commit mark instead, and accept the SHA without
checking (as it is expected to be in another repository).

> Oh, and your patch lacks test cases that demonstrate how you envisage the 
> whole thing to work.

Ok, I'll make some tests tomorrow. For now, this is an example of output from
my test repository:

...
blob
mark :16
data 41
[submodule "sub"]
        path = sub
        url = sub

commit refs/heads/master
mark :17
author Alexander Gavrilov <angavrilov@gmail.com> 1216360728 +0400
committer Alexander Gavrilov <angavrilov@gmail.com> 1216360728 +0400
data 4
sub
from :15
M 100644 :16 .gitmodules
M 160000 d6317bc6e2597bf74ae199514a54e25775b0d20d sub


Alexander

Re: [RFC PATCH] Support gitlinks in fast-import/export.

[Shawn O. Pearce]

Alexander Gavrilov <angavrilov@gmail.com> wrote:
> Currently fast-import/export cannot be used for
> repositories with submodules. This patch extends
> the relevant programs to make them correctly
> process gitlinks.
> 
> Signed-off-by: Alexander Gavrilov <angavrilov@gmail.com>

Acked-by: Shawn O. Pearce <spearce@spearce.org>

> 	I noticed that fast-export & fast-import cannot work with
> 	repositories using submodules: import complains about
> 	an invalid mode, and export fails while trying to open the SHA
> 	as a blob.
> 
> 	As I didn't see any particular reason for it to be so, I tried to
> 	implement support for gitlinks.
> 
> 	What I'm unsure of is, should fast-export try to reuse commit
> 	marks for gitlinks where it happened to recognize the object,
> 	or always output the SHA as it is stored in the tree?

Its unlikely that the commit objects are in the repository the
export is running in, so its unlikely you can use a mark.  Its
fine to just always output the SHA-1 I think.

-- 
Shawn.

Re: [RFC PATCH] Support gitlinks in fast-import/export.

[Johannes Schindelin]

Hi,

On Fri, 18 Jul 2008, Alexander Gavrilov wrote:

> On Friday 18 July 2008 21:36:26 Johannes Schindelin wrote:
> > > 	What I'm unsure of is, should fast-export try to reuse commit 
> > > 	marks for gitlinks where it happened to recognize the object, or 
> > > 	always output the SHA as it is stored in the tree?
>  
> > Are they commit marks?  No.  So they should be handled as marks, just 
> > as those for blobs and trees.
> > 
> > (They are commit marks in the _submodule_, but that does not matter 
> > here.)
> 
> Well, I was thinking of that unlikely case when the master module and 
> the submodule are in the same repository and are exported together. But 
> probably it is best to just output the SHA after all.

Exactly.

What you describe is a _possibility_.  Requiring it would be a serious 
mistake.

> > >  	for (i = 0; i < diff_queued_diff.nr; i++)
> > > -		handle_object(diff_queued_diff.queue[i]->two->sha1);
> > > +		if (!S_ISGITLINK(diff_queued_diff.queue[i]->two->mode))
> > > +			handle_object(diff_queued_diff.queue[i]->two->sha1);
> > 
> > Why?  You do not want to export changes in the submodules?
> 
> handle_object opens the sha as a blob and outputs it. As gitlinks aren't 
> blobs, it won't work. And if the submodule is in a separate repository, 
> there is nothing to open anyway.

So what should happen to them instead?  Ignoring them?

Possible.

My earlier remark about the marks was this: you might want to mark SHA-1s 
of gitlinks with a (shorter) number, but maybe that is just bullocks.  At 
the same time, it might be not.

> Simultaneously reading commits from the submodule repository is a whole 
> different level of complexity. I'm afraid I'm not up to it yet.

Good news for you:  I think it would be a serious mistake to read commits 
from the submodule.

> > >  		else {
> > >  			struct object *object = lookup_object(spec->sha1);
> > > -			printf("M %06o :%d %s\n", spec->mode,
> > > -			       get_object_mark(object), spec->path);
> > > +			int mark = object ? get_object_mark(object) : 0;
> > 
> > As I said, that looks wrong.  Maybe we have to fake objects for the 
> > gitlinks.
> 
> I tried to avoid faking blobs and stick to the interface of the M 
> command itself.

My point was: I do not see gitlinks handled _at all_.

I agree that they should not be handled the same as blobs, but i did not 
have time to check that gitlinks are not just ignored by your patch.  
Which would be wrong: you want the exported commits to contain them.

> > > @@ -1900,7 +1901,16 @@ static void file_change_m(struct branch *b)
> > >  		p = uq.buf;
> > >  	}
> > >  
> > > -	if (inline_data) {
> > > +	if (S_ISGITLINK(mode)) {
> > > +		if (inline_data)
> > > +			die("Git links cannot be specified 'inline': %s",
> > > +				command_buf.buf);
> > > +		else if (oe) {
> > > +			if (oe->type != OBJ_COMMIT)
> > > +				die("Not a commit (actually a %s): %s",
> > > +					typename(oe->type), command_buf.buf);
> > 
> > How is that supposed to work?  Do I understand correctly that you 
> > require the user to construct a commit object for the gitlink?  That 
> > would be actively wrong.
> 
> There are three forms of the M command:
> 
> M mode inline some/path
> ...some data...
> 
> M mode :mark some/path
> 
> M mode SHA some/path
> 
> For usual files the mark must be created by the 'blob' command,
> and the SHA must refer to an existing blob. This hunk makes fast-import
> require for gitlinks a commit mark instead, and accept the SHA without
> checking (as it is expected to be in another repository).

What's this commit mark thing?  I hope you do not mean to ask the user to 
construct a commit object in the _superproject's_ repository...

> > Oh, and your patch lacks test cases that demonstrate how you envisage 
> > the whole thing to work.
> 
> Ok, I'll make some tests tomorrow.

Maybe I should enhance on my point, to drive it home:

If you do _not_ include automated tests, other people have less reason to 
trust that your patch does what you insist it does.

If you do not include tests, and have a sparse commit message (which you 
do), people are left to guess what your patch tries to do, and do not even 
have the chance to see something you wanted to work.

If you do not include tests, and your patch does something a reviewer 
feels it should not to, or omits something a reviewer feels it _should_ 
do, there is less of an opportunity to see if this was intended: a 
comprehensive test script would show what the submitter expects to 
work alright.

So in a very real sense, it is advisable to write the test _first_, and 
then the patch.

Not everything XP brought to this world is evil.  (Oh yes, you guessed it, 
I was talking about eXtreme Programming...)

Ciao,
Dscho

Re: [RFC PATCH] Support gitlinks in fast-import/export.

[Johannes Schindelin]

Hi,

On Fri, 18 Jul 2008, Shawn O. Pearce wrote:

> Alexander Gavrilov <angavrilov@gmail.com> wrote:
> 
> > 	I noticed that fast-export & fast-import cannot work with 
> > 	repositories using submodules: import complains about an invalid 
> > 	mode, and export fails while trying to open the SHA as a blob.
> > 
> > 	As I didn't see any particular reason for it to be so, I tried to 
> > 	implement support for gitlinks.
> > 
> > 	What I'm unsure of is, should fast-export try to reuse commit 
> > 	marks for gitlinks where it happened to recognize the object, or 
> > 	always output the SHA as it is stored in the tree?
> 
> Its unlikely that the commit objects are in the repository the export is 
> running in, so its unlikely you can use a mark.  Its fine to just always 
> output the SHA-1 I think.

Oh, I just realized that fast-import expects changes, instead of full 
snapshots.

So strike all my suggestions regarding gitlink marks.

Sorry,
Dscho