[PATCH 0/4] Make unpack_trees() do separate source and destination indexes

[PATCH 0/4] Make unpack_trees() do separate source and destination indexes

[Linus Torvalds]

Ok, this goes on top of the five-commit series yesterday.  The first two
patches are pure fixes and cleanups, the third one does a fairly
mechanical thing to get rid of implied usage of "the_index", and the
last one actually splits up the source and destination index handling.

Linus Torvalds (4):
  Fix tree-walking compare_entry() in the presense of --prefix
  Add 'const' where appropriate to index handling functions
  Make 'unpack_trees()' take the index to work on as an argument
  Make 'unpack_trees()' have a separate source and destination index

The nice thing to see is how this clearly removes more lines than it
adds, and the code is a bit more readable too, I think.

 builtin-checkout.c        |    9 +++
 builtin-commit.c          |    2 +
 builtin-merge-recursive.c |    2 +
 builtin-read-tree.c       |   23 +------
 cache.h                   |   10 ++--
 diff-lib.c                |   47 ++-----------
 hash.c                    |    6 +-
 hash.h                    |    4 +-
 read-cache.c              |   12 ++--
 tree-walk.c               |    3 +
 unpack-trees.c            |  158 ++++++++++++++++++++++-----------------------
 unpack-trees.h            |   15 +++--
 12 files changed, 128 insertions(+), 163 deletions(-)

[PATCH 3/4] Make 'unpack_trees()' take the index to work on as an argument

[Linus Torvalds]

This is just a very mechanical conversion, and makes everybody set it to
'&the_index' before calling, but at least it makes it more explicit
where we work with the index.

The next stage would be to split that index usage up into a 'source' and
a 'destination' index, so that we can unpack into a different index than
we started out from.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
---
 builtin-checkout.c        |    6 +++
 builtin-commit.c          |    1 +
 builtin-merge-recursive.c |    1 +
 builtin-read-tree.c       |    1 +
 diff-lib.c                |    2 +
 unpack-trees.c            |   79 +++++++++++++++++++++++----------------------
 unpack-trees.h            |    1 +
 7 files changed, 52 insertions(+), 39 deletions(-)

diff --git a/builtin-checkout.c b/builtin-checkout.c
index 6b08016..9bdb623 100644
--- a/builtin-checkout.c
+++ b/builtin-checkout.c
@@ -152,6 +152,7 @@ static int reset_to_new(struct tree *tree, int quiet)
 {
 	struct unpack_trees_options opts;
 	struct tree_desc tree_desc;
+
 	memset(&opts, 0, sizeof(opts));
 	opts.head_idx = -1;
 	opts.update = 1;
@@ -159,6 +160,7 @@ static int reset_to_new(struct tree *tree, int quiet)
 	opts.merge = 1;
 	opts.fn = oneway_merge;
 	opts.verbose_update = !quiet;
+	opts.index = &the_index;
 	parse_tree(tree);
 	init_tree_desc(&tree_desc, tree->buffer, tree->size);
 	if (unpack_trees(1, &tree_desc, &opts))
@@ -170,6 +172,7 @@ static void reset_clean_to_new(struct tree *tree, int quiet)
 {
 	struct unpack_trees_options opts;
 	struct tree_desc tree_desc;
+
 	memset(&opts, 0, sizeof(opts));
 	opts.head_idx = -1;
 	opts.skip_unmerged = 1;
@@ -177,6 +180,7 @@ static void reset_clean_to_new(struct tree *tree, int quiet)
 	opts.merge = 1;
 	opts.fn = oneway_merge;
 	opts.verbose_update = !quiet;
+	opts.index = &the_index;
 	parse_tree(tree);
 	init_tree_desc(&tree_desc, tree->buffer, tree->size);
 	if (unpack_trees(1, &tree_desc, &opts))
@@ -224,8 +228,10 @@ static int merge_working_tree(struct checkout_opts *opts,
 		struct tree_desc trees[2];
 		struct tree *tree;
 		struct unpack_trees_options topts;
+
 		memset(&topts, 0, sizeof(topts));
 		topts.head_idx = -1;
+		topts.index = &the_index;
 
 		refresh_cache(REFRESH_QUIET);
 
diff --git a/builtin-commit.c b/builtin-commit.c
index f49c22e..38a5422 100644
--- a/builtin-commit.c
+++ b/builtin-commit.c
@@ -198,6 +198,7 @@ static void create_base_index(void)
 	opts.head_idx = 1;
 	opts.index_only = 1;
 	opts.merge = 1;
+	opts.index = &the_index;
 
 	opts.fn = oneway_merge;
 	tree = parse_tree_indirect(head_sha1);
diff --git a/builtin-merge-recursive.c b/builtin-merge-recursive.c
index 6fe4102..50b3896 100644
--- a/builtin-merge-recursive.c
+++ b/builtin-merge-recursive.c
@@ -213,6 +213,7 @@ static int git_merge_trees(int index_only,
 	opts.merge = 1;
 	opts.head_idx = 2;
 	opts.fn = threeway_merge;
+	opts.index = &the_index;
 
 	init_tree_desc_from_tree(t+0, common);
 	init_tree_desc_from_tree(t+1, head);
diff --git a/builtin-read-tree.c b/builtin-read-tree.c
index 0138f5a..d004e90 100644
--- a/builtin-read-tree.c
+++ b/builtin-read-tree.c
@@ -102,6 +102,7 @@ int cmd_read_tree(int argc, const char **argv, const char *unused_prefix)
 
 	memset(&opts, 0, sizeof(opts));
 	opts.head_idx = -1;
+	opts.index = &the_index;
 
 	git_config(git_default_config);
 
diff --git a/diff-lib.c b/diff-lib.c
index 4581b59..e359058 100644
--- a/diff-lib.c
+++ b/diff-lib.c
@@ -734,6 +734,7 @@ int run_diff_index(struct rev_info *revs, int cached)
 	opts.merge = 1;
 	opts.fn = oneway_diff;
 	opts.unpack_data = revs;
+	opts.index = &the_index;
 
 	init_tree_desc(&t, tree->buffer, tree->size);
 	if (unpack_trees(1, &t, &opts))
@@ -787,6 +788,7 @@ int do_diff_cache(const unsigned char *tree_sha1, struct diff_options *opt)
 	opts.merge = 1;
 	opts.fn = oneway_diff;
 	opts.unpack_data = &revs;
+	opts.index = &the_index;
 
 	init_tree_desc(&t, tree->buffer, tree->size);
 	if (unpack_trees(1, &t, &opts))
diff --git a/unpack-trees.c b/unpack-trees.c
index ee9be29..cb8f847 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -1,3 +1,4 @@
+#define NO_THE_INDEX_COMPATIBILITY_MACROS
 #include "cache.h"
 #include "dir.h"
 #include "tree.h"
@@ -7,10 +8,10 @@
 #include "progress.h"
 #include "refs.h"
 
-static inline void remove_entry(int remove)
+static inline void remove_entry(int remove, struct unpack_trees_options *o)
 {
 	if (remove >= 0)
-		remove_cache_entry_at(remove);
+		remove_index_entry_at(o->index, remove);
 }
 
 /* Unlink the last component and attempt to remove leading
@@ -53,8 +54,8 @@ static void check_updates(struct unpack_trees_options *o)
 	int i;
 
 	if (o->update && o->verbose_update) {
-		for (total = cnt = 0; cnt < active_nr; cnt++) {
-			struct cache_entry *ce = active_cache[cnt];
+		for (total = cnt = 0; cnt < o->index->cache_nr; cnt++) {
+			struct cache_entry *ce = o->index->cache[cnt];
 			if (ce->ce_flags & (CE_UPDATE | CE_REMOVE))
 				total++;
 		}
@@ -65,15 +66,15 @@ static void check_updates(struct unpack_trees_options *o)
 	}
 
 	*last_symlink = '\0';
-	for (i = 0; i < active_nr; i++) {
-		struct cache_entry *ce = active_cache[i];
+	for (i = 0; i < o->index->cache_nr; i++) {
+		struct cache_entry *ce = o->index->cache[i];
 
 		if (ce->ce_flags & (CE_UPDATE | CE_REMOVE))
 			display_progress(progress, ++cnt);
 		if (ce->ce_flags & CE_REMOVE) {
 			if (o->update)
 				unlink_entry(ce->name, last_symlink);
-			remove_cache_entry_at(i);
+			remove_index_entry_at(o->index, i);
 			i--;
 			continue;
 		}
@@ -105,7 +106,7 @@ static int unpack_index_entry(struct cache_entry *ce, struct unpack_trees_option
 		if (o->skip_unmerged) {
 			o->pos++;
 		} else {
-			remove_entry(o->pos);
+			remove_entry(o->pos, o);
 		}
 		return 0;
 	}
@@ -242,9 +243,9 @@ static int unpack_nondirectories(int n, unsigned long mask, unsigned long dirmas
 		return call_unpack_fn(src, o, remove);
 
 	n += o->merge;
-	remove_entry(remove);
+	remove_entry(remove, o);
 	for (i = 0; i < n; i++)
-		add_cache_entry(src[i], ADD_CACHE_OK_TO_ADD|ADD_CACHE_SKIP_DFCHECK);
+		add_index_entry(o->index, src[i], ADD_CACHE_OK_TO_ADD|ADD_CACHE_SKIP_DFCHECK);
 	return 0;
 }
 
@@ -261,8 +262,8 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str
 
 	/* Are we supposed to look at the index too? */
 	if (o->merge) {
-		while (o->pos < active_nr) {
-			struct cache_entry *ce = active_cache[o->pos];
+		while (o->pos < o->index->cache_nr) {
+			struct cache_entry *ce = o->index->cache[o->pos];
 			int cmp = compare_entry(ce, info, p);
 			if (cmp < 0) {
 				if (unpack_index_entry(ce, o) < 0)
@@ -277,7 +278,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str
 					 */
 					if (o->skip_unmerged)
 						return mask;
-					remove_entry(o->pos);
+					remove_entry(o->pos, o);
 					continue;
 				}
 				src[0] = ce;
@@ -312,8 +313,8 @@ static int unpack_failed(struct unpack_trees_options *o, const char *message)
 			return error(message);
 		return -1;
 	}
-	discard_cache();
-	read_cache();
+	discard_index(o->index);
+	read_index(o->index);
 	return -1;
 }
 
@@ -349,8 +350,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 
 	/* Any left-over entries in the index? */
 	if (o->merge) {
-		while (o->pos < active_nr) {
-			struct cache_entry *ce = active_cache[o->pos];
+		while (o->pos < o->index->cache_nr) {
+			struct cache_entry *ce = o->index->cache[o->pos];
 			if (unpack_index_entry(ce, o) < 0)
 				return unpack_failed(o, NULL);
 		}
@@ -395,7 +396,7 @@ static int verify_uptodate(struct cache_entry *ce,
 		return 0;
 
 	if (!lstat(ce->name, &st)) {
-		unsigned changed = ce_match_stat(ce, &st, CE_MATCH_IGNORE_VALID);
+		unsigned changed = ie_match_stat(o->index, ce, &st, CE_MATCH_IGNORE_VALID);
 		if (!changed)
 			return 0;
 		/*
@@ -415,10 +416,10 @@ static int verify_uptodate(struct cache_entry *ce,
 		error("Entry '%s' not uptodate. Cannot merge.", ce->name);
 }
 
-static void invalidate_ce_path(struct cache_entry *ce)
+static void invalidate_ce_path(struct cache_entry *ce, struct unpack_trees_options *o)
 {
 	if (ce)
-		cache_tree_invalidate_path(active_cache_tree, ce->name);
+		cache_tree_invalidate_path(o->index->cache_tree, ce->name);
 }
 
 /*
@@ -463,12 +464,12 @@ static int verify_clean_subdirectory(struct cache_entry *ce, const char *action,
 	 * in that directory.
 	 */
 	namelen = strlen(ce->name);
-	pos = cache_name_pos(ce->name, namelen);
+	pos = index_name_pos(o->index, ce->name, namelen);
 	if (0 <= pos)
 		return cnt; /* we have it as nondirectory */
 	pos = -pos - 1;
-	for (i = pos; i < active_nr; i++) {
-		struct cache_entry *ce = active_cache[i];
+	for (i = pos; i < o->index->cache_nr; i++) {
+		struct cache_entry *ce = o->index->cache[i];
 		int len = ce_namelen(ce);
 		if (len < namelen ||
 		    strncmp(ce->name, ce->name, namelen) ||
@@ -566,9 +567,9 @@ static int verify_absent(struct cache_entry *ce, const char *action,
 		 * delete this path, which is in a subdirectory that
 		 * is being replaced with a blob.
 		 */
-		cnt = cache_name_pos(ce->name, strlen(ce->name));
+		cnt = index_name_pos(o->index, ce->name, strlen(ce->name));
 		if (0 <= cnt) {
-			struct cache_entry *ce = active_cache[cnt];
+			struct cache_entry *ce = o->index->cache[cnt];
 			if (ce->ce_flags & CE_REMOVE)
 				return 0;
 		}
@@ -597,17 +598,17 @@ static int merged_entry(struct cache_entry *merge, struct cache_entry *old,
 		} else {
 			if (verify_uptodate(old, o))
 				return -1;
-			invalidate_ce_path(old);
+			invalidate_ce_path(old, o);
 		}
 	}
 	else {
 		if (verify_absent(merge, "overwritten", o))
 			return -1;
-		invalidate_ce_path(merge);
+		invalidate_ce_path(merge, o);
 	}
 
 	merge->ce_flags &= ~CE_STAGEMASK;
-	add_cache_entry(merge, ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE);
+	add_index_entry(o->index, merge, ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE);
 	return 1;
 }
 
@@ -621,14 +622,14 @@ static int deleted_entry(struct cache_entry *ce, struct cache_entry *old,
 		if (verify_absent(ce, "removed", o))
 			return -1;
 	ce->ce_flags |= CE_REMOVE;
-	add_cache_entry(ce, ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE);
-	invalidate_ce_path(ce);
+	add_index_entry(o->index, ce, ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE);
+	invalidate_ce_path(ce, o);
 	return 1;
 }
 
 static int keep_entry(struct cache_entry *ce, struct unpack_trees_options *o)
 {
-	add_cache_entry(ce, ADD_CACHE_OK_TO_ADD);
+	add_index_entry(o->index, ce, ADD_CACHE_OK_TO_ADD);
 	return 1;
 }
 
@@ -728,7 +729,7 @@ int threeway_merge(struct cache_entry **stages,
 
 	/* #1 */
 	if (!head && !remote && any_anc_missing) {
-		remove_entry(remove);
+		remove_entry(remove, o);
 		return 0;
 	}
 
@@ -762,7 +763,7 @@ int threeway_merge(struct cache_entry **stages,
 		if ((head_deleted && remote_deleted) ||
 		    (head_deleted && remote && remote_match) ||
 		    (remote_deleted && head && head_match)) {
-			remove_entry(remove);
+			remove_entry(remove, o);
 			if (index)
 				return deleted_entry(index, index, o);
 			else if (ce && !head_deleted) {
@@ -788,7 +789,7 @@ int threeway_merge(struct cache_entry **stages,
 			return -1;
 	}
 
-	remove_entry(remove);
+	remove_entry(remove, o);
 	o->nontrivial_merge = 1;
 
 	/* #2, #3, #4, #6, #7, #9, #10, #11. */
@@ -853,7 +854,7 @@ int twoway_merge(struct cache_entry **src,
 		}
 		else if (oldtree && !newtree && same(current, oldtree)) {
 			/* 10 or 11 */
-			remove_entry(remove);
+			remove_entry(remove, o);
 			return deleted_entry(oldtree, current, o);
 		}
 		else if (oldtree && newtree &&
@@ -863,7 +864,7 @@ int twoway_merge(struct cache_entry **src,
 		}
 		else {
 			/* all other failures */
-			remove_entry(remove);
+			remove_entry(remove, o);
 			if (oldtree)
 				return o->gently ? -1 : reject_merge(oldtree);
 			if (current)
@@ -875,7 +876,7 @@ int twoway_merge(struct cache_entry **src,
 	}
 	else if (newtree)
 		return merged_entry(newtree, current, o);
-	remove_entry(remove);
+	remove_entry(remove, o);
 	return deleted_entry(oldtree, current, o);
 }
 
@@ -922,14 +923,14 @@ int oneway_merge(struct cache_entry **src,
 			     o->merge_size);
 
 	if (!a) {
-		remove_entry(remove);
+		remove_entry(remove, o);
 		return deleted_entry(old, old, o);
 	}
 	if (old && same(old, a)) {
 		if (o->reset) {
 			struct stat st;
 			if (lstat(old->name, &st) ||
-			    ce_match_stat(old, &st, CE_MATCH_IGNORE_VALID))
+			    ie_match_stat(o->index, old, &st, CE_MATCH_IGNORE_VALID))
 				old->ce_flags |= CE_UPDATE;
 		}
 		return keep_entry(old, o);
diff --git a/unpack-trees.h b/unpack-trees.h
index a2df544..65add16 100644
--- a/unpack-trees.h
+++ b/unpack-trees.h
@@ -28,6 +28,7 @@ struct unpack_trees_options {
 
 	struct cache_entry *df_conflict_entry;
 	void *unpack_data;
+	struct index_state *index;
 };
 
 extern int unpack_trees(unsigned n, struct tree_desc *t,
-- 
1.5.4.3.452.g67136

[PATCH 2/4] Add 'const' where appropriate to index handling functions

[Linus Torvalds]

This is in an effort to make the source index of 'unpack_trees()' as
being const, and thus making the compiler help us verify that we only
access it for reading.

The constification also extended to some of the hashing helpers that get
called indirectly.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
---
 cache.h      |   10 +++++-----
 hash.c       |    6 +++---
 hash.h       |    4 ++--
 read-cache.c |   12 ++++++------
 4 files changed, 16 insertions(+), 16 deletions(-)

diff --git a/cache.h b/cache.h
index 6eb16cb..b519b7a 100644
--- a/cache.h
+++ b/cache.h
@@ -346,12 +346,12 @@ extern void verify_non_filename(const char *prefix, const char *name);
 /* Initialize and use the cache information */
 extern int read_index(struct index_state *);
 extern int read_index_from(struct index_state *, const char *path);
-extern int write_index(struct index_state *, int newfd);
+extern int write_index(const struct index_state *, int newfd);
 extern int discard_index(struct index_state *);
-extern int unmerged_index(struct index_state *);
+extern int unmerged_index(const struct index_state *);
 extern int verify_path(const char *path);
 extern int index_name_exists(struct index_state *istate, const char *name, int namelen);
-extern int index_name_pos(struct index_state *, const char *name, int namelen);
+extern int index_name_pos(const struct index_state *, const char *name, int namelen);
 #define ADD_CACHE_OK_TO_ADD 1		/* Ok to add */
 #define ADD_CACHE_OK_TO_REPLACE 2	/* Ok to replace file/directory */
 #define ADD_CACHE_SKIP_DFCHECK 4	/* Ok to skip DF conflict checks */
@@ -368,8 +368,8 @@ extern int ce_same_name(struct cache_entry *a, struct cache_entry *b);
 #define CE_MATCH_IGNORE_VALID		01
 /* do not check the contents but report dirty on racily-clean entries */
 #define CE_MATCH_RACY_IS_DIRTY	02
-extern int ie_match_stat(struct index_state *, struct cache_entry *, struct stat *, unsigned int);
-extern int ie_modified(struct index_state *, struct cache_entry *, struct stat *, unsigned int);
+extern int ie_match_stat(const struct index_state *, struct cache_entry *, struct stat *, unsigned int);
+extern int ie_modified(const struct index_state *, struct cache_entry *, struct stat *, unsigned int);
 
 extern int ce_path_match(const struct cache_entry *ce, const char **pathspec);
 extern int index_fd(unsigned char *sha1, int fd, struct stat *st, int write_object, enum object_type type, const char *path);
diff --git a/hash.c b/hash.c
index d9ec82f..1cd4c9d 100644
--- a/hash.c
+++ b/hash.c
@@ -9,7 +9,7 @@
  * the existing entry, or the empty slot if none existed. The caller
  * can then look at the (*ptr) to see whether it existed or not.
  */
-static struct hash_table_entry *lookup_hash_entry(unsigned int hash, struct hash_table *table)
+static struct hash_table_entry *lookup_hash_entry(unsigned int hash, const struct hash_table *table)
 {
 	unsigned int size = table->size, nr = hash % size;
 	struct hash_table_entry *array = table->array;
@@ -66,7 +66,7 @@ static void grow_hash_table(struct hash_table *table)
 	free(old_array);
 }
 
-void *lookup_hash(unsigned int hash, struct hash_table *table)
+void *lookup_hash(unsigned int hash, const struct hash_table *table)
 {
 	if (!table->array)
 		return NULL;
@@ -81,7 +81,7 @@ void **insert_hash(unsigned int hash, void *ptr, struct hash_table *table)
 	return insert_hash_entry(hash, ptr, table);
 }
 
-int for_each_hash(struct hash_table *table, int (*fn)(void *))
+int for_each_hash(const struct hash_table *table, int (*fn)(void *))
 {
 	int sum = 0;
 	unsigned int i;
diff --git a/hash.h b/hash.h
index a8b0fbb..69e33a4 100644
--- a/hash.h
+++ b/hash.h
@@ -28,9 +28,9 @@ struct hash_table {
 	struct hash_table_entry *array;
 };
 
-extern void *lookup_hash(unsigned int hash, struct hash_table *table);
+extern void *lookup_hash(unsigned int hash, const struct hash_table *table);
 extern void **insert_hash(unsigned int hash, void *ptr, struct hash_table *table);
-extern int for_each_hash(struct hash_table *table, int (*fn)(void *));
+extern int for_each_hash(const struct hash_table *table, int (*fn)(void *));
 extern void free_hash(struct hash_table *table);
 
 static inline void init_hash(struct hash_table *table)
diff --git a/read-cache.c b/read-cache.c
index bf649a3..a92b25b 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -255,13 +255,13 @@ static int ce_match_stat_basic(struct cache_entry *ce, struct stat *st)
 	return changed;
 }
 
-static int is_racy_timestamp(struct index_state *istate, struct cache_entry *ce)
+static int is_racy_timestamp(const struct index_state *istate, struct cache_entry *ce)
 {
 	return (istate->timestamp &&
 		((unsigned int)istate->timestamp) <= ce->ce_mtime);
 }
 
-int ie_match_stat(struct index_state *istate,
+int ie_match_stat(const struct index_state *istate,
 		  struct cache_entry *ce, struct stat *st,
 		  unsigned int options)
 {
@@ -304,7 +304,7 @@ int ie_match_stat(struct index_state *istate,
 	return changed;
 }
 
-int ie_modified(struct index_state *istate,
+int ie_modified(const struct index_state *istate,
 		struct cache_entry *ce, struct stat *st, unsigned int options)
 {
 	int changed, changed_fs;
@@ -412,7 +412,7 @@ int cache_name_compare(const char *name1, int flags1, const char *name2, int fla
 	return 0;
 }
 
-int index_name_pos(struct index_state *istate, const char *name, int namelen)
+int index_name_pos(const struct index_state *istate, const char *name, int namelen)
 {
 	int first, last;
 
@@ -1201,7 +1201,7 @@ int discard_index(struct index_state *istate)
 	return 0;
 }
 
-int unmerged_index(struct index_state *istate)
+int unmerged_index(const struct index_state *istate)
 {
 	int i;
 	for (i = 0; i < istate->cache_nr; i++) {
@@ -1346,7 +1346,7 @@ static int ce_write_entry(SHA_CTX *c, int fd, struct cache_entry *ce)
 	return ce_write(c, fd, ondisk, size);
 }
 
-int write_index(struct index_state *istate, int newfd)
+int write_index(const struct index_state *istate, int newfd)
 {
 	SHA_CTX c;
 	struct cache_header hdr;
-- 
1.5.4.3.452.g67136

[PATCH 1/4] Fix tree-walking compare_entry() in the presense of --prefix

[Linus Torvalds]

When we make the "root" tree-walk info entry have a pathname in it, we
need to have a ->prev pointer so that compare_entry will actually notice
and traverse into the root.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
---
 tree-walk.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/tree-walk.c b/tree-walk.c
index 842cb6a..02e2aed 100644
--- a/tree-walk.c
+++ b/tree-walk.c
@@ -107,6 +107,7 @@ int tree_entry(struct tree_desc *desc, struct name_entry *entry)
 void setup_traverse_info(struct traverse_info *info, const char *base)
 {
 	int pathlen = strlen(base);
+	static struct traverse_info dummy;
 
 	memset(info, 0, sizeof(*info));
 	if (pathlen && base[pathlen-1] == '/')
@@ -114,6 +115,8 @@ void setup_traverse_info(struct traverse_info *info, const char *base)
 	info->pathlen = pathlen ? pathlen + 1 : 0;
 	info->name.path = base;
 	info->name.sha1 = (void *)(base + pathlen + 1);
+	if (pathlen)
+		info->prev = &dummy;
 }
 
 char *make_traverse_path(char *path, const struct traverse_info *info, const struct name_entry *n)
-- 
1.5.4.3.452.g67136

[PATCH 4/4] Make 'unpack_trees()' have a separate source and destination index

[Linus Torvalds]

We will always unpack into our own internal index, but we will take the
source from wherever specified, and we will optionally write the result
to a specified index (optionally, because not everybody even _wants_ any
result: the index diffing really wants to just walk the tree and index
in parallel).

This ends up removing a fair number more lines than it adds, for the
simple reason that we can now skip all the crud that tried to be
oh-so-careful about maintaining our position in the index as we were
traversing and modifying it.  Since we don't actually modify the source
index any more, we can just update the 'o->pos' pointer without worrying
about whether an index entry got removed or replaced or added to.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
---
 builtin-checkout.c        |    9 ++-
 builtin-commit.c          |    3 +-
 builtin-merge-recursive.c |    3 +-
 builtin-read-tree.c       |   24 +-------
 diff-lib.c                |   49 +++------------
 unpack-trees.c            |  149 ++++++++++++++++++++++-----------------------
 unpack-trees.h            |   16 +++--
 7 files changed, 101 insertions(+), 152 deletions(-)

diff --git a/builtin-checkout.c b/builtin-checkout.c
index 9bdb623..7deb504 100644
--- a/builtin-checkout.c
+++ b/builtin-checkout.c
@@ -160,7 +160,8 @@ static int reset_to_new(struct tree *tree, int quiet)
 	opts.merge = 1;
 	opts.fn = oneway_merge;
 	opts.verbose_update = !quiet;
-	opts.index = &the_index;
+	opts.src_index = &the_index;
+	opts.dst_index = &the_index;
 	parse_tree(tree);
 	init_tree_desc(&tree_desc, tree->buffer, tree->size);
 	if (unpack_trees(1, &tree_desc, &opts))
@@ -180,7 +181,8 @@ static void reset_clean_to_new(struct tree *tree, int quiet)
 	opts.merge = 1;
 	opts.fn = oneway_merge;
 	opts.verbose_update = !quiet;
-	opts.index = &the_index;
+	opts.src_index = &the_index;
+	opts.dst_index = &the_index;
 	parse_tree(tree);
 	init_tree_desc(&tree_desc, tree->buffer, tree->size);
 	if (unpack_trees(1, &tree_desc, &opts))
@@ -231,7 +233,8 @@ static int merge_working_tree(struct checkout_opts *opts,
 
 		memset(&topts, 0, sizeof(topts));
 		topts.head_idx = -1;
-		topts.index = &the_index;
+		topts.src_index = &the_index;
+		topts.dst_index = &the_index;
 
 		refresh_cache(REFRESH_QUIET);
 
diff --git a/builtin-commit.c b/builtin-commit.c
index 38a5422..660a345 100644
--- a/builtin-commit.c
+++ b/builtin-commit.c
@@ -198,7 +198,8 @@ static void create_base_index(void)
 	opts.head_idx = 1;
 	opts.index_only = 1;
 	opts.merge = 1;
-	opts.index = &the_index;
+	opts.src_index = &the_index;
+	opts.dst_index = &the_index;
 
 	opts.fn = oneway_merge;
 	tree = parse_tree_indirect(head_sha1);
diff --git a/builtin-merge-recursive.c b/builtin-merge-recursive.c
index 50b3896..fa02bb5 100644
--- a/builtin-merge-recursive.c
+++ b/builtin-merge-recursive.c
@@ -213,7 +213,8 @@ static int git_merge_trees(int index_only,
 	opts.merge = 1;
 	opts.head_idx = 2;
 	opts.fn = threeway_merge;
-	opts.index = &the_index;
+	opts.src_index = &the_index;
+	opts.dst_index = &the_index;
 
 	init_tree_desc_from_tree(t+0, common);
 	init_tree_desc_from_tree(t+1, head);
diff --git a/builtin-read-tree.c b/builtin-read-tree.c
index d004e90..160456d 100644
--- a/builtin-read-tree.c
+++ b/builtin-read-tree.c
@@ -102,7 +102,8 @@ int cmd_read_tree(int argc, const char **argv, const char *unused_prefix)
 
 	memset(&opts, 0, sizeof(opts));
 	opts.head_idx = -1;
-	opts.index = &the_index;
+	opts.src_index = &the_index;
+	opts.dst_index = &the_index;
 
 	git_config(git_default_config);
 
@@ -221,27 +222,6 @@ int cmd_read_tree(int argc, const char **argv, const char *unused_prefix)
 	if ((opts.dir && !opts.update))
 		die("--exclude-per-directory is meaningless unless -u");
 
-	if (opts.prefix) {
-		int pfxlen = strlen(opts.prefix);
-		int pos;
-		if (opts.prefix[pfxlen-1] != '/')
-			die("prefix must end with /");
-		if (stage != 2)
-			die("binding merge takes only one tree");
-		pos = cache_name_pos(opts.prefix, pfxlen);
-		if (0 <= pos)
-			die("corrupt index file");
-		pos = -pos-1;
-		if (pos < active_nr &&
-		    !strncmp(active_cache[pos]->name, opts.prefix, pfxlen))
-			die("subdirectory '%s' already exists.", opts.prefix);
-		pos = cache_name_pos(opts.prefix, pfxlen-1);
-		if (0 <= pos)
-			die("file '%.*s' already exists.",
-					pfxlen-1, opts.prefix);
-		opts.pos = -1 - pos;
-	}
-
 	if (opts.merge) {
 		if (stage < 2)
 			die("just how do you expect me to merge %d trees?", stage-1);
diff --git a/diff-lib.c b/diff-lib.c
index e359058..9520773 100644
--- a/diff-lib.c
+++ b/diff-lib.c
@@ -600,8 +600,7 @@ static void mark_merge_entries(void)
  */
 static void do_oneway_diff(struct unpack_trees_options *o,
 	struct cache_entry *idx,
-	struct cache_entry *tree,
-	int idx_pos, int idx_nr)
+	struct cache_entry *tree)
 {
 	struct rev_info *revs = o->unpack_data;
 	int match_missing, cached;
@@ -643,34 +642,6 @@ static void do_oneway_diff(struct unpack_trees_options *o,
 }
 
 /*
- * Count how many index entries go with the first one
- */
-static inline int count_skip(const struct cache_entry *src, int pos)
-{
-	int skip = 1;
-
-	/* We can only have multiple entries if the first one is not stage-0 */
-	if (ce_stage(src)) {
-		struct cache_entry **p = active_cache + pos;
-		int namelen = ce_namelen(src);
-
-		for (;;) {
-			const struct cache_entry *ce;
-			pos++;
-			if (pos >= active_nr)
-				break;
-			ce = *++p;
-			if (ce_namelen(ce) != namelen)
-				break;
-			if (memcmp(ce->name, src->name, namelen))
-				break;
-			skip++;
-		}
-	}
-	return skip;
-}
-
-/*
  * The unpack_trees() interface is designed for merging, so
  * the different source entries are designed primarily for
  * the source trees, with the old index being really mainly
@@ -685,18 +656,12 @@ static inline int count_skip(const struct cache_entry *src, int pos)
  * the fairly complex unpack_trees() semantic requirements, including
  * the skipping, the path matching, the type conflict cases etc.
  */
-static int oneway_diff(struct cache_entry **src,
-	struct unpack_trees_options *o,
-	int index_pos)
+static int oneway_diff(struct cache_entry **src, struct unpack_trees_options *o)
 {
-	int skip = 0;
 	struct cache_entry *idx = src[0];
 	struct cache_entry *tree = src[1];
 	struct rev_info *revs = o->unpack_data;
 
-	if (index_pos >= 0)
-		skip = count_skip(idx, index_pos);
-
 	/*
 	 * Unpack-trees generates a DF/conflict entry if
 	 * there was a directory in the index and a tree
@@ -707,9 +672,9 @@ static int oneway_diff(struct cache_entry **src,
 		tree = NULL;
 
 	if (ce_path_match(idx ? idx : tree, revs->prune_data))
-		do_oneway_diff(o, idx, tree, index_pos, skip);
+		do_oneway_diff(o, idx, tree);
 
-	return skip;
+	return 0;
 }
 
 int run_diff_index(struct rev_info *revs, int cached)
@@ -734,7 +699,8 @@ int run_diff_index(struct rev_info *revs, int cached)
 	opts.merge = 1;
 	opts.fn = oneway_diff;
 	opts.unpack_data = revs;
-	opts.index = &the_index;
+	opts.src_index = &the_index;
+	opts.dst_index = NULL;
 
 	init_tree_desc(&t, tree->buffer, tree->size);
 	if (unpack_trees(1, &t, &opts))
@@ -788,7 +754,8 @@ int do_diff_cache(const unsigned char *tree_sha1, struct diff_options *opt)
 	opts.merge = 1;
 	opts.fn = oneway_diff;
 	opts.unpack_data = &revs;
-	opts.index = &the_index;
+	opts.src_index = &the_index;
+	opts.dst_index = &the_index;
 
 	init_tree_desc(&t, tree->buffer, tree->size);
 	if (unpack_trees(1, &t, &opts))
diff --git a/unpack-trees.c b/unpack-trees.c
index cb8f847..0cdf198 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -8,10 +8,18 @@
 #include "progress.h"
 #include "refs.h"
 
-static inline void remove_entry(int remove, struct unpack_trees_options *o)
+static void add_entry(struct unpack_trees_options *o, struct cache_entry *ce,
+	unsigned int set, unsigned int clear)
 {
-	if (remove >= 0)
-		remove_index_entry_at(o->index, remove);
+	unsigned int size = ce_size(ce);
+	struct cache_entry *new = xmalloc(size);
+
+	clear |= CE_HASHED | CE_UNHASHED;
+
+	memcpy(new, ce, size);
+	new->next = NULL;
+	new->ce_flags = (new->ce_flags & ~clear) | set;
+	add_index_entry(&o->result, new, ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE|ADD_CACHE_SKIP_DFCHECK);
 }
 
 /* Unlink the last component and attempt to remove leading
@@ -51,11 +59,12 @@ static void check_updates(struct unpack_trees_options *o)
 	unsigned cnt = 0, total = 0;
 	struct progress *progress = NULL;
 	char last_symlink[PATH_MAX];
+	struct index_state *index = &o->result;
 	int i;
 
 	if (o->update && o->verbose_update) {
-		for (total = cnt = 0; cnt < o->index->cache_nr; cnt++) {
-			struct cache_entry *ce = o->index->cache[cnt];
+		for (total = cnt = 0; cnt < index->cache_nr; cnt++) {
+			struct cache_entry *ce = index->cache[cnt];
 			if (ce->ce_flags & (CE_UPDATE | CE_REMOVE))
 				total++;
 		}
@@ -66,15 +75,15 @@ static void check_updates(struct unpack_trees_options *o)
 	}
 
 	*last_symlink = '\0';
-	for (i = 0; i < o->index->cache_nr; i++) {
-		struct cache_entry *ce = o->index->cache[i];
+	for (i = 0; i < index->cache_nr; i++) {
+		struct cache_entry *ce = index->cache[i];
 
 		if (ce->ce_flags & (CE_UPDATE | CE_REMOVE))
 			display_progress(progress, ++cnt);
 		if (ce->ce_flags & CE_REMOVE) {
 			if (o->update)
 				unlink_entry(ce->name, last_symlink);
-			remove_index_entry_at(o->index, i);
+			remove_index_entry_at(&o->result, i);
 			i--;
 			continue;
 		}
@@ -89,28 +98,27 @@ static void check_updates(struct unpack_trees_options *o)
 	stop_progress(&progress);
 }
 
-static inline int call_unpack_fn(struct cache_entry **src, struct unpack_trees_options *o, int remove)
+static inline int call_unpack_fn(struct cache_entry **src, struct unpack_trees_options *o)
 {
-	int ret = o->fn(src, o, remove);
-	if (ret > 0) {
-		o->pos += ret;
+	int ret = o->fn(src, o);
+	if (ret > 0)
 		ret = 0;
-	}
 	return ret;
 }
 
 static int unpack_index_entry(struct cache_entry *ce, struct unpack_trees_options *o)
 {
 	struct cache_entry *src[5] = { ce, };
+
+	o->pos++;
 	if (ce_stage(ce)) {
 		if (o->skip_unmerged) {
-			o->pos++;
-		} else {
-			remove_entry(o->pos, o);
+			add_entry(o, ce, 0, 0);
+			return 0;
 		}
 		return 0;
 	}
-	return call_unpack_fn(src, o, o->pos);
+	return call_unpack_fn(src, o);
 }
 
 int traverse_trees_recursive(int n, unsigned long dirmask, unsigned long df_conflicts, struct name_entry *names, struct traverse_info *info)
@@ -200,7 +208,7 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info, con
 }
 
 static int unpack_nondirectories(int n, unsigned long mask, unsigned long dirmask, struct cache_entry *src[5],
-	const struct name_entry *names, const struct traverse_info *info, int remove)
+	const struct name_entry *names, const struct traverse_info *info)
 {
 	int i;
 	struct unpack_trees_options *o = info->data;
@@ -240,12 +248,11 @@ static int unpack_nondirectories(int n, unsigned long mask, unsigned long dirmas
 	}
 
 	if (o->merge)
-		return call_unpack_fn(src, o, remove);
+		return call_unpack_fn(src, o);
 
 	n += o->merge;
-	remove_entry(remove, o);
 	for (i = 0; i < n; i++)
-		add_index_entry(o->index, src[i], ADD_CACHE_OK_TO_ADD|ADD_CACHE_SKIP_DFCHECK);
+		add_entry(o, src[i], 0, 0);
 	return 0;
 }
 
@@ -253,7 +260,6 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str
 {
 	struct cache_entry *src[5] = { NULL, };
 	struct unpack_trees_options *o = info->data;
-	int remove = -1;
 	const struct name_entry *p = names;
 
 	/* Find first entry with a real name (we could use "mask" too) */
@@ -262,8 +268,8 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str
 
 	/* Are we supposed to look at the index too? */
 	if (o->merge) {
-		while (o->pos < o->index->cache_nr) {
-			struct cache_entry *ce = o->index->cache[o->pos];
+		while (o->pos < o->src_index->cache_nr) {
+			struct cache_entry *ce = o->src_index->cache[o->pos];
 			int cmp = compare_entry(ce, info, p);
 			if (cmp < 0) {
 				if (unpack_index_entry(ce, o) < 0)
@@ -271,24 +277,25 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str
 				continue;
 			}
 			if (!cmp) {
+				o->pos++;
 				if (ce_stage(ce)) {
 					/*
 					 * If we skip unmerged index entries, we'll skip this
 					 * entry *and* the tree entries associated with it!
 					 */
-					if (o->skip_unmerged)
+					if (o->skip_unmerged) {
+						add_entry(o, ce, 0, 0);
 						return mask;
-					remove_entry(o->pos, o);
+					}
 					continue;
 				}
 				src[0] = ce;
-				remove = o->pos;
 			}
 			break;
 		}
 	}
 
-	if (unpack_nondirectories(n, mask, dirmask, src, names, info, remove) < 0)
+	if (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)
 		return -1;
 
 	/* Now handle any directories.. */
@@ -313,8 +320,6 @@ static int unpack_failed(struct unpack_trees_options *o, const char *message)
 			return error(message);
 		return -1;
 	}
-	discard_index(o->index);
-	read_index(o->index);
 	return -1;
 }
 
@@ -330,6 +335,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 	state.quiet = 1;
 	state.refresh_cache = 1;
 
+	memset(&o->result, 0, sizeof(o->result));
 	o->merge_size = len;
 
 	if (!dfc)
@@ -350,8 +356,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 
 	/* Any left-over entries in the index? */
 	if (o->merge) {
-		while (o->pos < o->index->cache_nr) {
-			struct cache_entry *ce = o->index->cache[o->pos];
+		while (o->pos < o->src_index->cache_nr) {
+			struct cache_entry *ce = o->src_index->cache[o->pos];
 			if (unpack_index_entry(ce, o) < 0)
 				return unpack_failed(o, NULL);
 		}
@@ -360,7 +366,10 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 	if (o->trivial_merges_only && o->nontrivial_merge)
 		return unpack_failed(o, "Merge requires file-level merging");
 
+	o->src_index = NULL;
 	check_updates(o);
+	if (o->dst_index)
+		*o->dst_index = o->result;
 	return 0;
 }
 
@@ -396,7 +405,7 @@ static int verify_uptodate(struct cache_entry *ce,
 		return 0;
 
 	if (!lstat(ce->name, &st)) {
-		unsigned changed = ie_match_stat(o->index, ce, &st, CE_MATCH_IGNORE_VALID);
+		unsigned changed = ie_match_stat(o->src_index, ce, &st, CE_MATCH_IGNORE_VALID);
 		if (!changed)
 			return 0;
 		/*
@@ -419,7 +428,7 @@ static int verify_uptodate(struct cache_entry *ce,
 static void invalidate_ce_path(struct cache_entry *ce, struct unpack_trees_options *o)
 {
 	if (ce)
-		cache_tree_invalidate_path(o->index->cache_tree, ce->name);
+		cache_tree_invalidate_path(o->src_index->cache_tree, ce->name);
 }
 
 /*
@@ -464,12 +473,12 @@ static int verify_clean_subdirectory(struct cache_entry *ce, const char *action,
 	 * in that directory.
 	 */
 	namelen = strlen(ce->name);
-	pos = index_name_pos(o->index, ce->name, namelen);
+	pos = index_name_pos(o->src_index, ce->name, namelen);
 	if (0 <= pos)
 		return cnt; /* we have it as nondirectory */
 	pos = -pos - 1;
-	for (i = pos; i < o->index->cache_nr; i++) {
-		struct cache_entry *ce = o->index->cache[i];
+	for (i = pos; i < o->src_index->cache_nr; i++) {
+		struct cache_entry *ce = o->src_index->cache[i];
 		int len = ce_namelen(ce);
 		if (len < namelen ||
 		    strncmp(ce->name, ce->name, namelen) ||
@@ -481,7 +490,7 @@ static int verify_clean_subdirectory(struct cache_entry *ce, const char *action,
 		if (!ce_stage(ce)) {
 			if (verify_uptodate(ce, o))
 				return -1;
-			ce->ce_flags |= CE_REMOVE;
+			add_entry(o, ce, CE_REMOVE, 0);
 		}
 		cnt++;
 	}
@@ -567,9 +576,9 @@ static int verify_absent(struct cache_entry *ce, const char *action,
 		 * delete this path, which is in a subdirectory that
 		 * is being replaced with a blob.
 		 */
-		cnt = index_name_pos(o->index, ce->name, strlen(ce->name));
+		cnt = index_name_pos(&o->result, ce->name, strlen(ce->name));
 		if (0 <= cnt) {
-			struct cache_entry *ce = o->index->cache[cnt];
+			struct cache_entry *ce = o->result.cache[cnt];
 			if (ce->ce_flags & CE_REMOVE)
 				return 0;
 		}
@@ -584,7 +593,6 @@ static int verify_absent(struct cache_entry *ce, const char *action,
 static int merged_entry(struct cache_entry *merge, struct cache_entry *old,
 		struct unpack_trees_options *o)
 {
-	merge->ce_flags |= CE_UPDATE;
 	if (old) {
 		/*
 		 * See if we can re-use the old CE directly?
@@ -607,29 +615,29 @@ static int merged_entry(struct cache_entry *merge, struct cache_entry *old,
 		invalidate_ce_path(merge, o);
 	}
 
-	merge->ce_flags &= ~CE_STAGEMASK;
-	add_index_entry(o->index, merge, ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE);
+	add_entry(o, merge, CE_UPDATE, CE_STAGEMASK);
 	return 1;
 }
 
 static int deleted_entry(struct cache_entry *ce, struct cache_entry *old,
 		struct unpack_trees_options *o)
 {
-	if (old) {
-		if (verify_uptodate(old, o))
-			return -1;
-	} else
+	/* Did it exist in the index? */
+	if (!old) {
 		if (verify_absent(ce, "removed", o))
 			return -1;
-	ce->ce_flags |= CE_REMOVE;
-	add_index_entry(o->index, ce, ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE);
+		return 0;
+	}
+	if (verify_uptodate(old, o))
+		return -1;
+	add_entry(o, ce, CE_REMOVE, 0);
 	invalidate_ce_path(ce, o);
 	return 1;
 }
 
 static int keep_entry(struct cache_entry *ce, struct unpack_trees_options *o)
 {
-	add_index_entry(o->index, ce, ADD_CACHE_OK_TO_ADD);
+	add_entry(o, ce, 0, 0);
 	return 1;
 }
 
@@ -649,9 +657,7 @@ static void show_stage_entry(FILE *o,
 }
 #endif
 
-int threeway_merge(struct cache_entry **stages,
-		struct unpack_trees_options *o,
-		int remove)
+int threeway_merge(struct cache_entry **stages, struct unpack_trees_options *o)
 {
 	struct cache_entry *index;
 	struct cache_entry *head;
@@ -728,10 +734,8 @@ int threeway_merge(struct cache_entry **stages,
 	}
 
 	/* #1 */
-	if (!head && !remote && any_anc_missing) {
-		remove_entry(remove, o);
+	if (!head && !remote && any_anc_missing)
 		return 0;
-	}
 
 	/* Under the new "aggressive" rule, we resolve mostly trivial
 	 * cases that we historically had git-merge-one-file resolve.
@@ -763,10 +767,9 @@ int threeway_merge(struct cache_entry **stages,
 		if ((head_deleted && remote_deleted) ||
 		    (head_deleted && remote && remote_match) ||
 		    (remote_deleted && head && head_match)) {
-			remove_entry(remove, o);
 			if (index)
 				return deleted_entry(index, index, o);
-			else if (ce && !head_deleted) {
+			if (ce && !head_deleted) {
 				if (verify_absent(ce, "removed", o))
 					return -1;
 			}
@@ -789,7 +792,6 @@ int threeway_merge(struct cache_entry **stages,
 			return -1;
 	}
 
-	remove_entry(remove, o);
 	o->nontrivial_merge = 1;
 
 	/* #2, #3, #4, #6, #7, #9, #10, #11. */
@@ -824,9 +826,7 @@ int threeway_merge(struct cache_entry **stages,
  * "carry forward" rule, please see <Documentation/git-read-tree.txt>.
  *
  */
-int twoway_merge(struct cache_entry **src,
-		struct unpack_trees_options *o,
-		int remove)
+int twoway_merge(struct cache_entry **src, struct unpack_trees_options *o)
 {
 	struct cache_entry *current = src[0];
 	struct cache_entry *oldtree = src[1];
@@ -854,7 +854,6 @@ int twoway_merge(struct cache_entry **src,
 		}
 		else if (oldtree && !newtree && same(current, oldtree)) {
 			/* 10 or 11 */
-			remove_entry(remove, o);
 			return deleted_entry(oldtree, current, o);
 		}
 		else if (oldtree && newtree &&
@@ -864,7 +863,6 @@ int twoway_merge(struct cache_entry **src,
 		}
 		else {
 			/* all other failures */
-			remove_entry(remove, o);
 			if (oldtree)
 				return o->gently ? -1 : reject_merge(oldtree);
 			if (current)
@@ -876,7 +874,6 @@ int twoway_merge(struct cache_entry **src,
 	}
 	else if (newtree)
 		return merged_entry(newtree, current, o);
-	remove_entry(remove, o);
 	return deleted_entry(oldtree, current, o);
 }
 
@@ -887,8 +884,7 @@ int twoway_merge(struct cache_entry **src,
  * stage0 does not have anything there.
  */
 int bind_merge(struct cache_entry **src,
-		struct unpack_trees_options *o,
-		int remove)
+		struct unpack_trees_options *o)
 {
 	struct cache_entry *old = src[0];
 	struct cache_entry *a = src[1];
@@ -898,7 +894,7 @@ int bind_merge(struct cache_entry **src,
 			     o->merge_size);
 	if (a && old)
 		return o->gently ? -1 :
-			error("Entry '%s' overlaps.  Cannot bind.", a->name);
+			error("Entry '%s' overlaps with '%s'.  Cannot bind.", a->name, old->name);
 	if (!a)
 		return keep_entry(old, o);
 	else
@@ -911,9 +907,7 @@ int bind_merge(struct cache_entry **src,
  * The rule is:
  * - take the stat information from stage0, take the data from stage1
  */
-int oneway_merge(struct cache_entry **src,
-		struct unpack_trees_options *o,
-		int remove)
+int oneway_merge(struct cache_entry **src, struct unpack_trees_options *o)
 {
 	struct cache_entry *old = src[0];
 	struct cache_entry *a = src[1];
@@ -922,18 +916,19 @@ int oneway_merge(struct cache_entry **src,
 		return error("Cannot do a oneway merge of %d trees",
 			     o->merge_size);
 
-	if (!a) {
-		remove_entry(remove, o);
+	if (!a)
 		return deleted_entry(old, old, o);
-	}
+
 	if (old && same(old, a)) {
+		int update = 0;
 		if (o->reset) {
 			struct stat st;
 			if (lstat(old->name, &st) ||
-			    ie_match_stat(o->index, old, &st, CE_MATCH_IGNORE_VALID))
-				old->ce_flags |= CE_UPDATE;
+			    ie_match_stat(o->src_index, old, &st, CE_MATCH_IGNORE_VALID))
+				update |= CE_UPDATE;
 		}
-		return keep_entry(old, o);
+		add_entry(o, old, update, 0);
+		return 0;
 	}
 	return merged_entry(a, old, o);
 }
diff --git a/unpack-trees.h b/unpack-trees.h
index 65add16..e8abbcd 100644
--- a/unpack-trees.h
+++ b/unpack-trees.h
@@ -4,8 +4,7 @@
 struct unpack_trees_options;
 
 typedef int (*merge_fn_t)(struct cache_entry **src,
-		struct unpack_trees_options *options,
-		int remove);
+		struct unpack_trees_options *options);
 
 struct unpack_trees_options {
 	int reset;
@@ -28,15 +27,18 @@ struct unpack_trees_options {
 
 	struct cache_entry *df_conflict_entry;
 	void *unpack_data;
-	struct index_state *index;
+
+	struct index_state *dst_index;
+	const struct index_state *src_index;
+	struct index_state result;
 };
 
 extern int unpack_trees(unsigned n, struct tree_desc *t,
 		struct unpack_trees_options *options);
 
-int threeway_merge(struct cache_entry **stages, struct unpack_trees_options *o, int);
-int twoway_merge(struct cache_entry **src, struct unpack_trees_options *o, int);
-int bind_merge(struct cache_entry **src, struct unpack_trees_options *o, int);
-int oneway_merge(struct cache_entry **src, struct unpack_trees_options *o, int);
+int threeway_merge(struct cache_entry **stages, struct unpack_trees_options *o);
+int twoway_merge(struct cache_entry **src, struct unpack_trees_options *o);
+int bind_merge(struct cache_entry **src, struct unpack_trees_options *o);
+int oneway_merge(struct cache_entry **src, struct unpack_trees_options *o);
 
 #endif
-- 
1.5.4.3.452.g67136

Re: [PATCH 4/4] Make 'unpack_trees()' have a separate source and destination index

[Johannes Schindelin]

Hi,

On Thu, 6 Mar 2008, Linus Torvalds wrote:

> We will always unpack into our own internal index, but we will take the
> source from wherever specified, and we will optionally write the result
> to a specified index (optionally, because not everybody even _wants_ any
> result: the index diffing really wants to just walk the tree and index
> in parallel).
> 
> This ends up removing a fair number more lines than it adds, for the 
> simple reason that we can now skip all the crud that tried to be 
> oh-so-careful about maintaining our position in the index as we were 
> traversing and modifying it.  Since we don't actually modify the source 
> index any more, we can just update the 'o->pos' pointer without worrying 
> about whether an index entry got removed or replaced or added to.

Nice.  Although you could have done better on the account of remove/add 
ratio if you did not add empty lines ;-)  But then, you removed a few 
curly brackets around single lines, too ;-)

> @@ -221,27 +222,6 @@ int cmd_read_tree(int argc, const char **argv, const char *unused_prefix)
>  	if ((opts.dir && !opts.update))
>  		die("--exclude-per-directory is meaningless unless -u");
>  
> -	if (opts.prefix) {
> -		int pfxlen = strlen(opts.prefix);
> -		int pos;
> -		if (opts.prefix[pfxlen-1] != '/')
> -			die("prefix must end with /");
> -		if (stage != 2)
> -			die("binding merge takes only one tree");
> -		pos = cache_name_pos(opts.prefix, pfxlen);
> -		if (0 <= pos)
> -			die("corrupt index file");
> -		pos = -pos-1;
> -		if (pos < active_nr &&
> -		    !strncmp(active_cache[pos]->name, opts.prefix, pfxlen))
> -			die("subdirectory '%s' already exists.", opts.prefix);
> -		pos = cache_name_pos(opts.prefix, pfxlen-1);
> -		if (0 <= pos)
> -			die("file '%.*s' already exists.",
> -					pfxlen-1, opts.prefix);
> -		opts.pos = -1 - pos;
> -	}
> -

Was the wholesale removal intentional?  I think there are a few sanity 
checks, and I did not see the checks moved to somewhere else.  But then, 
there could be redundant checks somewhere else that I missed.

> @@ -360,7 +366,10 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
>  	if (o->trivial_merges_only && o->nontrivial_merge)
>  		return unpack_failed(o, "Merge requires file-level merging");
>  
> +	o->src_index = NULL;
>  	check_updates(o);
> +	if (o->dst_index)
> +		*o->dst_index = o->result;
>  	return 0;
>  }
>  

I wonder if you should discard_index(o->dst_index) if o->src_index == 
o->dst_index (before you set it to NULL, of course).

I cannot really comment on the rest, since the whole unpack_trees() logic 
was too complicated for me.  Maybe I have a chance after this patch.

Ciao,
Dscho

Re: [PATCH 4/4] Make 'unpack_trees()' have a separate source and destination index

[Linus Torvalds]

On Fri, 7 Mar 2008, Johannes Schindelin wrote:
> 
> > @@ -221,27 +222,6 @@ int cmd_read_tree(int argc, const char **argv, const char *unused_prefix)
> >  	if ((opts.dir && !opts.update))
> >  		die("--exclude-per-directory is meaningless unless -u");
> >  
> > -	if (opts.prefix) {
> > -		int pfxlen = strlen(opts.prefix);
> > -		int pos;
> > -		if (opts.prefix[pfxlen-1] != '/')
> > -			die("prefix must end with /");
> > -		if (stage != 2)
> > -			die("binding merge takes only one tree");
> > -		pos = cache_name_pos(opts.prefix, pfxlen);
> > -		if (0 <= pos)
> > -			die("corrupt index file");
> > -		pos = -pos-1;
> > -		if (pos < active_nr &&
> > -		    !strncmp(active_cache[pos]->name, opts.prefix, pfxlen))
> > -			die("subdirectory '%s' already exists.", opts.prefix);
> > -		pos = cache_name_pos(opts.prefix, pfxlen-1);
> > -		if (0 <= pos)
> > -			die("file '%.*s' already exists.",
> > -					pfxlen-1, opts.prefix);
> > -		opts.pos = -1 - pos;
> > -	}
> > -
> 
> Was the wholesale removal intentional?  I think there are a few sanity 
> checks, and I did not see the checks moved to somewhere else.  But then, 
> there could be redundant checks somewhere else that I missed.

It was intentional. The "bind_merge()" function already checks for overlap 
in the individual entries, so those sanity checks don't actually buy you 
anything. And those games with "opt.pos" actualyl made the new model not 
work, because it would mean that we'd skip the old entries before that 
"pos" entirely (instead of moving them over to the new index).

However, you're right to point it out. The old checks are somewhat 
different from the new ones - "bind_merge()" will check for overlap in 
individual entries, but it doesn't disallow merging non-overlapping 
subdirectories. I thought that extending the functionality would be a 
feature, but maybe somebody really wants those checks to be there on 
purpose.

So maybe we could have left a few of those checks. I just thought it was 
pretty interesting how we disallowed doing

	- index:
		subdirectory/Makefile

	- tree:
		.. anything *but* a 'Makefile' entry ..

	- git-read-tree --prefix subdirectory/ <tree>

and without the checks it actually works fine (it creates a "union" of the 
index and the incoming tree).

If somebody doesn't want that, I think they should check up-front (easy 
enough: just test if "git ls-files subdirectory/" returns anything), so I 
actually think the sanity checks just remove capabilities rather than add 
anything.

> > @@ -360,7 +366,10 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
> >  	if (o->trivial_merges_only && o->nontrivial_merge)
> >  		return unpack_failed(o, "Merge requires file-level merging");
> >  
> > +	o->src_index = NULL;
> >  	check_updates(o);
> > +	if (o->dst_index)
> > +		*o->dst_index = o->result;
> >  	return 0;
> >  }
> >  
> 
> I wonder if you should discard_index(o->dst_index) if o->src_index == 
> o->dst_index (before you set it to NULL, of course).

Not quite yet. I _really_ wanted to, but we can't do it - there's a few 
users that want to use "the_index" that I was not ready to work out what 
the right thing to do was.

In particular, when we do write_entry() of the finished index, that one 
does:

    check_updates
      checkout_entry()
        write_entry
          convert_to_working_tree
            git_checkattr
              bootstrap_attr_stack
                read_attr
                  read_index_data
                    index_name_pos(&the_index)

and the thing is, if we discard the old "the_index()" early (like we 
*should* do), that callchain breaks because the attribute code will use 
"the_index" rather than the index we are actually using!

Now, the fact is, I think that's a bug in the attribute code (well, 
"historical artifact" from the fact that we always used to use just 
"the_index" for everything). It should no longer (in my opinion) just 
blindly use "the_index" when we are writing out something else than 
"the_index", but hey, that's what it does.

If you want to play with it, here's a patch that I did *not* post on top 
of the whole series that makes the issue more obvious. That "memset()" is 
just to poison any users of the index that isn't finalized yet, to make 
you get a nice SIGSEGV rather than just inexplicable failures.

Anyway, to fix that thing, we'd have to pass the correct index around all 
the way, which is definitely worth doing regardless, but it was kind of 
beyond the aim of _this_ particular patch series.

Here's the patch to get you started :)

		Linus

---
 unpack-trees.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/unpack-trees.c b/unpack-trees.c
index 0cdf198..afa9c9d 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -366,6 +366,10 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 	if (o->trivial_merges_only && o->nontrivial_merge)
 		return unpack_failed(o, "Merge requires file-level merging");
 
+	if (o->dst_index) {
+		discard_index(o->dst_index);
+		memset(o->dst_index, 0x55, sizeof(*o->dst_index));
+	}
 	o->src_index = NULL;
 	check_updates(o);
 	if (o->dst_index)

Re: [PATCH 4/4] Make 'unpack_trees()' have a separate source and destination index

[Daniel Barkalow]

On Thu, 6 Mar 2008, Linus Torvalds wrote:

> We will always unpack into our own internal index, but we will take the
> source from wherever specified, and we will optionally write the result
> to a specified index (optionally, because not everybody even _wants_ any
> result: the index diffing really wants to just walk the tree and index
> in parallel).
> 
> This ends up removing a fair number more lines than it adds, for the
> simple reason that we can now skip all the crud that tried to be
> oh-so-careful about maintaining our position in the index as we were
> traversing and modifying it.  Since we don't actually modify the source
> index any more, we can just update the 'o->pos' pointer without worrying
> about whether an index entry got removed or replaced or added to.

It looks to me like it's leaking stuff stored in the index it creates if 
it ends up failing. I'm not entirely sure of the index lifecycle stuff, 
but it seems like it would be necessary. Aside from that, I think it 
should be right, although I haven't really gone over the resulting code in 
detail.

	-Daniel
*This .sig left intentionally blank*

Re: [PATCH 4/4] Make 'unpack_trees()' have a separate source and destination index

[Linus Torvalds]

On Fri, 7 Mar 2008, Daniel Barkalow wrote:
>
> It looks to me like it's leaking stuff stored in the index it creates if 
> it ends up failing.

Yes, adding a "discard_index(&o->result)" to the failure path sounds like 
a good idea. Patch appended.

> I'm not entirely sure of the index lifecycle stuff, 
> but it seems like it would be necessary.

Well, "necessary" may not be the right word, because it's a purely 
internal index thing, so all that happens is a memory leak. But it's 
certainly easy to do and correct.

That said, the *big* memory leak comes from the fact that we leak the 
cache_entry things left and right. That's a very traditional leak: because 
we used to build up the cache entries by just mapping them directly in 
from the index file (and we emulate that in modern times by allocating 
them from one big array), we can't actually free them one-by-one.

So doing the "discard_index()" will free the hash tables etc, which is 
good, and it will free the "istate->alloc" but that is never set on the 
result because we don't get the result from the index read. So we don't 
actually free the individual cache entries themselves that got created 
from the trees.

That's not something new, btw. We never did. But some day we should just 
add a flag to the cache_entry() that it's a "free one by one" kind, and 
then we could/should do it. In the meantime, this one-liner will fix 
*some* of the memory leaks, but not that old traditional one.

		Linus

---
 unpack-trees.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/unpack-trees.c b/unpack-trees.c
index 0cdf198..da68557 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -315,6 +315,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str
 
 static int unpack_failed(struct unpack_trees_options *o, const char *message)
 {
+	discard_index(&o->result);
 	if (!o->gently) {
 		if (message)
 			return error(message);

Re: [PATCH 0/4] Make unpack_trees() do separate source and destination indexes

[Junio C Hamano]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> Ok, this goes on top of the five-commit series yesterday.  The first two
> patches are pure fixes and cleanups, the third one does a fairly
> mechanical thing to get rid of implied usage of "the_index", and the
> last one actually splits up the source and destination index handling.

What I have been wondering about this series was if we can re-enable
cache-tree for more of the unpack_trees() users.

Currently, all unpack_trees() users, other than a single-tree read-tree,
invalidates the whole cache-tree information, as Daniel's "pop one, decide
what to put back, all in the original index" had too many manual index
manipulations and sprinkling cache_tree_invalidate() call everywhere was
too much cluttering of already hard-to-follow codepath.

I'd want to see at least two-way read-tree preserve as much cache-tree
information as possible.  People often work on one branch to completion
(i.e. make a commit, which involves a write-tree, which leaves a fully
valid cache-tree in the index), switch branches (i.e. two-way read-tree,
which unfortunately decimates cache-tree in the current implementation) to
commit a small fix.  If major parts of the cache-tree were left intact
during branch switching, these unchanged subdirectories do not have to be
rehashed in write-tree when making this final commit.

Re: [PATCH 0/4] Make unpack_trees() do separate source and destination indexes

[Linus Torvalds]

On Thu, 6 Mar 2008, Junio C Hamano wrote:
> 
> What I have been wondering about this series was if we can re-enable
> cache-tree for more of the unpack_trees() users.
> 
> Currently, all unpack_trees() users, other than a single-tree read-tree,
> invalidates the whole cache-tree information, as Daniel's "pop one, decide
> what to put back, all in the original index" had too many manual index
> manipulations and sprinkling cache_tree_invalidate() call everywhere was
> too much cluttering of already hard-to-follow codepath.

It should work as-is, but not optimally.

The reason I say that is that the result-tree won't have any cache tree 
*at*all* with this series, which is a possible peformance poblem (but 
there are performance upsides to the series too, so I'm not sure it's all 
that noticeable). So the "source" index will keep its cache-tree 
unmodified, because it simply never got modified in the first place.

Of course, all _current_ users will have src_index == dst_index (except 
for the "diff_oneway" thing that doesn't have a destination at all, 
because it doesn't actually want any index manipulation), so for all 
current users this is all pure downside: we throw away the good cache_tree 
for the incoming index, and we aren't even able to use it for the result.

But because I envisioned that the people who really *care* will be the 
ones that want to have different incoming and outgoing indexes, I didn't 
actually worry about it. Ie all the "git status" and "git commit" stuff 
would presumably end up using different indexes, and then they can re-use 
the (unmodified) source, and that would generally be the one that they 
actually want.

But I dunno. Maybe we'd actually want to try to re-use the cache_tree from 
the source some way, if we normally actually want it in the destination.

So this series really isn't a final thing, but I do think that in the form 
I posted it, it's a step forward in (a) readability/maintainability and 
(b) flexibility. And I think all the really *core* stuff got converted, 
but now there's a lot of small details hanging around that just don't take 
advantage of the new capabilities.

I basically tried to keep things bug-for-bug compatible. I also didn't try 
to optimize some things that I know I'll want to eventually.

For example, since we now create a new "index_state" entirely, that means 
that we may need to re-hash the cache entries in the name hash, and that 
means that we now allocate a new one *and*copy*it* rather than just re-use 
cache entries directly.

So that's the reason for the new add_entry() function that actually 
allocates a new cache_entry. And it actually has room for various very 
obvious optimizations that I did *not* do:

 - if the cache_entry comes from a tree (common), just re-use it, since it 
   wasn't hashed in the first place
 - if "o->dst_index" is NULL, just don't do anything, because we'll not 
   use the result anyway.
 - if "o->dst_index == o->src_index" and the cache_entry doesn't have 
   CE_HASHED set, just re-use it directly.

adn things like that. So there were a lot of small micro-optimizations 
that I didn't do, because quite frankly, this conversion was really 
painful in the first place (the set of patches I sent out may have 
_looked_ fairly clean, but that's only because I didn't show you the pain 
it was to create them ;)

		Linus