[PATCH 1/2] Teach git list-objects logic not to follow gitlinks

[PATCH 1/2] Teach git list-objects logic not to follow gitlinks

[Linus Torvalds]

This allows us to pack superprojects and thus clone them (but not yet
check them out on the receiving side - that's the next patch)

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
---

Most of this diff is a comment saying why there is nothing to do ;)

 list-objects.c |   34 ++++++++++++++++++++++++++++++++++
 1 files changed, 34 insertions(+), 0 deletions(-)

diff --git a/list-objects.c b/list-objects.c
index 2ba2c95..310f8d3 100644
--- a/list-objects.c
+++ b/list-objects.c
@@ -25,6 +25,37 @@ static void process_blob(struct rev_info *revs,
 	add_object(obj, p, path, name);
 }
 
+/*
+ * Processing a gitlink entry currently does nothing, since
+ * we do not recurse into the subproject.
+ *
+ * We *could* eventually add a flag that actually does that,
+ * which would involve:
+ *  - is the subproject actually checked out?
+ *  - if so, see if the subproject has already been added
+ *    to the alternates list, and add it if not.
+ *  - process the commit (or tag) the gitlink points to
+ *    recursively.
+ *
+ * However, it's unclear whether there is really ever any
+ * reason to see superprojects and subprojects as such a
+ * "unified" object pool (potentially resulting in a totally
+ * humongous pack - avoiding which was the whole point of
+ * having gitlinks in the first place!).
+ *
+ * So for now, there is just a note that we *could* follow
+ * the link, and how to do it. Whether it necessarily makes
+ * any sense what-so-ever to ever do that is another issue.
+ */
+static void process_gitlink(struct rev_info *revs,
+			    const unsigned char *sha1,
+			    struct object_array *p,
+			    struct name_path *path,
+			    const char *name)
+{
+	/* Nothing to do */
+}
+
 static void process_tree(struct rev_info *revs,
 			 struct tree *tree,
 			 struct object_array *p,
@@ -56,6 +87,9 @@ static void process_tree(struct rev_info *revs,
 			process_tree(revs,
 				     lookup_tree(entry.sha1),
 				     p, &me, entry.path);
+		else if (S_ISDIRLNK(entry.mode))
+			process_gitlink(revs, entry.sha1,
+					p, &me, entry.path);
 		else
 			process_blob(revs,
 				     lookup_blob(entry.sha1),

[PATCH 2/2] Teach "git-read-tree -u" to check out submodules as a directory

[Linus Torvalds]

This actually allows us to check out a supermodule after cloning, although 
the submodules will obviously not be checked out, and will just be an 
empty subdirectory.

[ Side note: this also shows that we currently don't correctly handle
  such subprojects that aren't checked out correctly yet.  They should
  always show up as not being modified, but failing to resolve the
  gitlink HEAD does not properly trigger the "not modified" logic in all 
  places it needs to..

  So more work to be done, but that's a separate issue, unrelated to
  the action of checking out the superproject. ]

The bulk of this patch is simply because we need to check the type of the 
index entry *before* we try to read the object it points to, and that 
meant that the code needed some re-organization. So I moved some of the 
code in common to both symlinks and files to be a trivial helper function.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
---
 entry.c |   42 +++++++++++++++++++++++++++++-------------
 1 files changed, 29 insertions(+), 13 deletions(-)

diff --git a/entry.c b/entry.c
index d72f811..9545e89 100644
--- a/entry.c
+++ b/entry.c
@@ -62,26 +62,33 @@ static int create_file(const char *path, unsigned int mode)
 	return open(path, O_WRONLY | O_CREAT | O_EXCL, mode);
 }
 
+static void *read_blob_entry(struct cache_entry *ce, const char *path, unsigned long *size)
+{
+	enum object_type type;
+	void *new = read_sha1_file(ce->sha1, &type, size);
+
+	if (new) {
+		if (type == OBJ_BLOB)
+			return new;
+		free(new);
+	}
+	return NULL;
+}
+
 static int write_entry(struct cache_entry *ce, char *path, struct checkout *state, int to_tempfile)
 {
 	int fd;
-	void *new;
-	unsigned long size;
 	long wrote;
-	enum object_type type;
 
-	new = read_sha1_file(ce->sha1, &type, &size);
-	if (!new || type != OBJ_BLOB) {
-		if (new)
-			free(new);
-		return error("git-checkout-index: unable to read sha1 file of %s (%s)",
-			path, sha1_to_hex(ce->sha1));
-	}
 	switch (ntohl(ce->ce_mode) & S_IFMT) {
-		char *buf;
-		unsigned long nsize;
+		char *buf, *new;
+		unsigned long size, nsize;
 
 	case S_IFREG:
+		new = read_blob_entry(ce, path, &size);
+		if (!new)
+			return error("git-checkout-index: unable to read sha1 file of %s (%s)",
+				path, sha1_to_hex(ce->sha1));
 		if (to_tempfile) {
 			strcpy(path, ".merge_file_XXXXXX");
 			fd = mkstemp(path);
@@ -111,6 +118,10 @@ static int write_entry(struct cache_entry *ce, char *path, struct checkout *stat
 			return error("git-checkout-index: unable to write file %s", path);
 		break;
 	case S_IFLNK:
+		new = read_blob_entry(ce, path, &size);
+		if (!new)
+			return error("git-checkout-index: unable to read sha1 file of %s (%s)",
+				path, sha1_to_hex(ce->sha1));
 		if (to_tempfile || !has_symlinks) {
 			if (to_tempfile) {
 				strcpy(path, ".merge_link_XXXXXX");
@@ -136,8 +147,13 @@ static int write_entry(struct cache_entry *ce, char *path, struct checkout *stat
 						 "symlink %s (%s)", path, strerror(errno));
 		}
 		break;
+	case S_IFDIRLNK:
+		if (to_tempfile)
+			return error("git-checkout-index: cannot create temporary subproject %s", path);
+		if (mkdir(path, 0777) < 0)
+			return error("git-checkout-index: cannot create subproject directory %s", path);
+		break;
 	default:
-		free(new);
 		return error("git-checkout-index: unknown file mode for %s", path);
 	}
 

Re: [PATCH 2/2] Teach "git-read-tree -u" to check out submodules as a directory

[Linus Torvalds]

On Thu, 12 Apr 2007, Linus Torvalds wrote:
> 
> [ Side note: this also shows that we currently don't correctly handle
>   such subprojects that aren't checked out correctly yet.  They should
>   always show up as not being modified, but failing to resolve the
>   gitlink HEAD does not properly trigger the "not modified" logic in all 
>   places it needs to..
> 
>   So more work to be done, but that's a separate issue, unrelated to
>   the action of checking out the superproject. ]

Apart from this issue, and the fact that we don't actually diff 
subprojects at all right now (even to the point of saying "subproject XyZ 
has changed from commit X to commit Y" - the raw format knows to say that, 
but the patch format does not), I think the "really core plumbing" is 
actually mostly done with this series.

So there's certainly some loose ends to tie up and some missing 
functionality even at the core level, but I suspect this is getting to the 
point where a big chunk of the remaining part is actually to teach the 
upper layers (like "fetch", "checkout", "merge" etc) to actually iterate 
over subprojects using the theoretical higher-level ".subprojects" file, 
and do the plumbing-level operations on those.

Of course, actually trying to start to *use* these things is bound to find 
tons and tons of issues, but I still feel like a lot of the core is at 
least "fleshed out", if not necessarily actually usable yet.

I may take a few days off on this, and hope others will step in and fix up 
some of the remaining problems ..

			Linus

Re: [PATCH 2/2] Teach "git-read-tree -u" to check out submodules as a directory

[Junio C Hamano]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> @@ -136,8 +147,13 @@ static int write_entry(struct cache_entry *ce, char *path, struct checkout *stat
>  						 "symlink %s (%s)", path, strerror(errno));
>  		}
>  		break;
> +	case S_IFDIRLNK:
> +		if (to_tempfile)
> +			return error("git-checkout-index: cannot create temporary subproject %s", path);
> +		if (mkdir(path, 0777) < 0)
> +			return error("git-checkout-index: cannot create subproject directory %s", path);
> +		break;
>  	default:
>  		return error("git-checkout-index: unknown file mode for %s", path);
>  	}
>  

Hmm.  Perhaps something like this on top?

diff --git a/entry.c b/entry.c
index 9545e89..0874d61 100644
--- a/entry.c
+++ b/entry.c
@@ -1,6 +1,23 @@
 #include "cache.h"
 #include "blob.h"
 
+static int make_directory(const char *path, int unlink_as_needed)
+{
+	if (mkdir(path, 0777) < 0) {
+		if (errno == EEXIST) {
+			struct stat st;
+			if (unlink_as_needed &&
+			    !unlink(path) &&
+			    !mkdir(path, 0777))
+				return 0; /* ok */
+			if (!stat(path, &st) && S_ISDIR(st.st_mode))
+				return 0; /* ok */
+		}
+		return -1;
+	}
+	return 0;
+}
+
 static void create_directories(const char *path, struct checkout *state)
 {
 	int len = strlen(path);
@@ -8,19 +25,14 @@ static void create_directories(const char *path, struct checkout *state)
 	const char *slash = path;
 
 	while ((slash = strchr(slash+1, '/')) != NULL) {
+		int unlink_as_needed;
+
 		len = slash - path;
 		memcpy(buf, path, len);
 		buf[len] = 0;
-		if (mkdir(buf, 0777)) {
-			if (errno == EEXIST) {
-				struct stat st;
-				if (len > state->base_dir_len && state->force && !unlink(buf) && !mkdir(buf, 0777))
-					continue;
-				if (!stat(buf, &st) && S_ISDIR(st.st_mode))
-					continue; /* ok */
-			}
+		unlink_as_needed = (state->base_dir_len < len && state->force);
+		if (make_directory(buf, unlink_as_needed) < 0)
 			die("cannot create directory at %s", buf);
-		}
 	}
 	free(buf);
 }
@@ -150,7 +162,7 @@ static int write_entry(struct cache_entry *ce, char *path, struct checkout *stat
 	case S_IFDIRLNK:
 		if (to_tempfile)
 			return error("git-checkout-index: cannot create temporary subproject %s", path);
-		if (mkdir(path, 0777) < 0)
+		if (make_directory(path, 0))
 			return error("git-checkout-index: cannot create subproject directory %s", path);
 		break;
 	default:

Re: [PATCH 2/2] Teach "git-read-tree -u" to check out submodules as a directory

[Linus Torvalds]

On Thu, 12 Apr 2007, Junio C Hamano wrote:
> 
> Hmm.  Perhaps something like this on top?
> 
> diff --git a/entry.c b/entry.c
> index 9545e89..0874d61 100644
> --- a/entry.c
> +++ b/entry.c
> @@ -1,6 +1,23 @@
>  #include "cache.h"
>  #include "blob.h"
>  
> +static int make_directory(const char *path, int unlink_as_needed)

I actually tried to think this through, and I don't *think* we should need 
to do this. It should literally be sufficient to just do the "mkdir()" 
call.

Why? Because we've already done the "create_directories()" calls earlier, 
and if you actually look at "create_file()", that one also just does a 
simple "open(path, O_WRONLY | O_CREAT | O_EXCL, mode)" - so doing just the 
mkdir() for creating a subdirectory should be the logically equivalent 
operation.

Similarly, the S_IFLNK case just does a "symlink()" system call. No "try 
to unlink if there was an old entry there before" code.

Basically, once we are inside "write_entry()", we expect to get a success 
or a failure, not a "needs cleanup". If it needed resolving of other state 
before, we shouldn't even have gotten to that stage in the first place.

That said, looking closer, you're right - the cleanup that is done earlier 
is actually wrong for the gitlink case. So I think the *real* problem is 
the cleanup in "checkout_entry()". In particular, note the

	if (!lstat(path, &st)) {
		...
		if (S_ISDIR(st.st_mode)) {
			...
		}
	}

code. If there was a directory there, we should actually leave it alone 
for the gitlink case, because it is up to the *subproject* checkout (not 
the superproject checkout) to handle any issues within that 
subdirectory!

So I think the *right* patch (on top of the one I send out) is actually 
just this:

		Linus

---
diff --git a/entry.c b/entry.c
index 9545e89..50ffae4 100644
--- a/entry.c
+++ b/entry.c
@@ -195,6 +195,9 @@ int checkout_entry(struct cache_entry *ce, struct checkout *state, char *topath)
 		 */
 		unlink(path);
 		if (S_ISDIR(st.st_mode)) {
+			/* If it is a gitlink, leave it alone! */
+			if (S_ISDIRLNK(ntohl(ce->ce_mode)))
+				return 0;
 			if (!state->force)
 				return error("%s is a directory", path);
 			remove_subtree(path);