[PATCH 0/8] Sparse checkout, the last half of the series

[PATCH 0/8] Sparse checkout, the last half of the series

[Nguyễn Thái Ngọc Duy]

This series is almost identical to my last sent series with bug fixes.
Documentation is hopefully detail enough so I don't have to say more here.

Nguyễn Thái Ngọc Duy (8):
  generate-cmdlist.sh: avoid selecting synopsis at wrong place
  Introduce "sparse patterns"
  unpack_trees(): keep track of unmerged entries
  unpack_trees(): add support for sparse checkout
  clone: support sparse checkout with --sparse-checkout option
  checkout: add new options to support sparse checkout
  Introduce default sparse patterns (core.defaultsparse)
  wt-status: show sparse checkout info

 Documentation/config.txt         |   10 +
 Documentation/git-checkout.txt   |  133 ++++++++++++++-
 Documentation/git-clone.txt      |   11 +-
 Documentation/git-ls-files.txt   |    8 +-
 builtin-checkout.c               |   93 ++++++++++
 builtin-clone.c                  |   16 ++
 builtin-ls-files.c               |   23 ++-
 builtin-read-tree.c              |    3 +-
 cache.h                          |    5 +
 config.c                         |    5 +
 environment.c                    |    1 +
 generate-cmdlist.sh              |    2 +-
 t/t2011-checkout-sparse.sh       |  147 ++++++++++++++++
 t/t3003-ls-files-narrow-match.sh |   41 +++++
 t/t3003/1                        |    4 +
 t/t3003/12                       |    8 +
 t/t3003/clone-escape             |    5 +
 t/t3003/cur-12                   |    2 +
 t/t3003/prefix-sub2              |    3 +
 t/t3003/root-sub-1               |    1 +
 t/t3003/slash-1                  |    1 +
 t/t3003/sub-1                    |    2 +
 t/t3003/sub-only                 |    3 +
 t/t3003/subsub-slash             |    3 +
 t/t5703-clone-narrow.sh          |   42 +++++
 unpack-trees.c                   |  360 +++++++++++++++++++++++++++++++++++++-
 unpack-trees.h                   |   29 +++
 wt-status.c                      |   44 +++++
 wt-status.h                      |    1 +
 29 files changed, 988 insertions(+), 18 deletions(-)
 create mode 100755 t/t2011-checkout-sparse.sh
 create mode 100755 t/t3003-ls-files-narrow-match.sh
 create mode 100644 t/t3003/1
 create mode 100644 t/t3003/12
 create mode 100644 t/t3003/clone-escape
 create mode 100644 t/t3003/cur-12
 create mode 100644 t/t3003/prefix-sub2
 create mode 100644 t/t3003/root-sub-1
 create mode 100644 t/t3003/slash-1
 create mode 100644 t/t3003/sub
 create mode 100644 t/t3003/sub-1
 create mode 100644 t/t3003/sub-only
 create mode 100644 t/t3003/subsub-slash
 create mode 100755 t/t5703-clone-narrow.sh

[PATCH 1/8] generate-cmdlist.sh: avoid selecting synopsis at wrong place

[Nguyễn Thái Ngọc Duy]

In "common" man pages there is luckily no "NAME" anywhere except at
beginning of documents. If there is another "NAME", sed could
mis-select it and lead to common-cmds.h corruption. So better nail it
at beginning of line, which would reduce corruption chance.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 generate-cmdlist.sh |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/generate-cmdlist.sh b/generate-cmdlist.sh
index a2913c2..75c68d9 100755
--- a/generate-cmdlist.sh
+++ b/generate-cmdlist.sh
@@ -14,7 +14,7 @@ sort |
 while read cmd
 do
      sed -n '
-     /NAME/,/git-'"$cmd"'/H
+     /^NAME/,/git-'"$cmd"'/H
      ${
             x
             s/.*git-'"$cmd"' - \(.*\)/  {"'"$cmd"'", "\1"},/
-- 
1.6.0.3.890.g95457

[PATCH 2/8] Introduce "sparse patterns"

[Nguyễn Thái Ngọc Duy]

Sparse patterns are basically .gitignore patterns, with the following
exceptions:

 - a leading slash means toplevel working directory
 - ./ will be used if you want to apply "slash in pattern" rule,
   similar to leading slash case in .gitignore
 - prefix can be specified in form prefix//pattern
 - patterns are separated by colons

"git ls-files --narrow-match" is added to test sparse patterns.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-checkout.txt   |   57 +++++++++++++
 Documentation/git-ls-files.txt   |    8 ++-
 builtin-ls-files.c               |   23 +++++-
 t/t3003-ls-files-narrow-match.sh |   41 +++++++++
 t/t3003/1                        |    4 +
 t/t3003/12                       |    8 ++
 t/t3003/clone-escape             |    5 +
 t/t3003/cur-12                   |    2 +
 t/t3003/prefix-sub2              |    3 +
 t/t3003/root-sub-1               |    1 +
 t/t3003/slash-1                  |    1 +
 t/t3003/sub-1                    |    2 +
 t/t3003/sub-only                 |    3 +
 t/t3003/subsub-slash             |    3 +
 unpack-trees.c                   |  170 ++++++++++++++++++++++++++++++++++++++
 unpack-trees.h                   |   20 +++++
 16 files changed, 347 insertions(+), 4 deletions(-)
 create mode 100755 t/t3003-ls-files-narrow-match.sh
 create mode 100644 t/t3003/1
 create mode 100644 t/t3003/12
 create mode 100644 t/t3003/clone-escape
 create mode 100644 t/t3003/cur-12
 create mode 100644 t/t3003/prefix-sub2
 create mode 100644 t/t3003/root-sub-1
 create mode 100644 t/t3003/slash-1
 create mode 100644 t/t3003/sub
 create mode 100644 t/t3003/sub-1
 create mode 100644 t/t3003/sub-only
 create mode 100644 t/t3003/subsub-slash

diff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt
index ea05a7a..0813d9f 100644
--- a/Documentation/git-checkout.txt
+++ b/Documentation/git-checkout.txt
@@ -205,6 +205,63 @@ is "assume unchanged" bit just ignores corresponding files in working
 directory while sparse checkout goes a bit farther, remove those files
 when it is safe to do so.
 
+Sparse patterns
+---------------
+
+Sparse patterns specify how do you want to form your checkout area.
+Many patterns can be specified on one line, separated by colons.
+The patterns specify what files should or should not be checked out
+on working directory (depends on the option used with the patterns).
+Patterns have the following format:
+
+ - An optional prefix '!' which negates the pattern; any
+   matching file by a previous pattern will become
+   unmatched again.  If a negated pattern matches, this will
+   override lower precedence patterns sources.
+
+ - If the pattern ends with a slash, it is removed for the
+   purpose of the following description, but it would only find
+   a match with a directory.  In other words, `foo/` will match a
+   directory `foo` and paths underneath it, but will not match a
+   regular file or a symbolic link `foo` (this is consistent
+   with the way how pathspec works in general in git).
+
+ - If the pattern does not contain a slash '/', git treats it as
+   a shell glob pattern and checks for a match against the
+   pathname without leading directories.
+
+ - If the pattern contains two consecutive slashes '//', git uses
+   the first part of the pattern before '//' as prefix to apply
+   the pattern on, and the rest as "true" sparse pattern. The
+   pattern will be only applied if pathname has the same prefix. It
+   is the same as you apply the pattern with the current directory
+   being the extracted prefix. Prefix will always be relative to
+   top-level working directory. This could be used to if you want to
+   apply the above rule only in a specific subdirectory.
+
+ - Otherwise, git treats the pattern as a shell glob suitable
+   for consumption by fnmatch(3) with the FNM_PATHNAME flag:
+   wildcards in the pattern will not match a / in the pathname.
+   For example, "Documentation/\*.html" matches
+   "Documentation/git.html" but not
+   "Documentation/ppc/ppc.html".  A leading slash matches the
+   beginning of the pathname; for example, "/*.c" matches
+   "cat-file.c" but not "mozilla-sha1/sha1.c".
+
+ - Patterns with a leading slash will match against full pathname,
+   as opposed to normal case when it only matches pathnames relative
+   to current working directory.
+
+ - Patterns begin with "./" are treated like normal patterns. That is
+   it will follow above rules. But since it has a slash inside,
+   "fnmatch rule" will apply. This is a work-around when you do not
+   want to apply "no slash" rule.
+
+ - Because colons are used to separate patterns, you cannot put them
+   in patterns directly. You must quote them using backslash.
+
+ - Sparse patterns do not apply to .gitignore and .gitattributes
+   files. They are always checked out.
 
 EXAMPLES
 --------
diff --git a/Documentation/git-ls-files.txt b/Documentation/git-ls-files.txt
index 1de68e2..fbed73b 100644
--- a/Documentation/git-ls-files.txt
+++ b/Documentation/git-ls-files.txt
@@ -12,7 +12,7 @@ SYNOPSIS
 'git ls-files' [-z] [-t] [-v]
 		(--[cached|deleted|others|ignored|stage|unmerged|killed|modified|orphaned|no-checkout])\*
 		(-[c|d|o|i|s|u|k|m])\*
-		[--sparse]
+		[--sparse] [--narrow-match=<sparse patterns>]
 		[-x <pattern>|--exclude=<pattern>]
 		[-X <file>|--exclude-from=<file>]
 		[--exclude-per-directory=<file>]
@@ -90,6 +90,12 @@ OPTIONS
 	No-checkout entries can be shown using --orphaned or
 	--no-checkout (or both).
 
+--narrow-match=<sparse patterns>::
+	This option can be used to test sparse patterns. The given sparse patterns will
+	be used to filter ls-files output. Entries not matching the spec will be
+	ignored. This option can only be used with --cached or --stage.
+	See linkgit:git-checkout[1] for more information about sparse patterns.
+
 -z::
 	\0 line termination on output.
 
diff --git a/builtin-ls-files.c b/builtin-ls-files.c
index 7606ce1..1b67ae2 100644
--- a/builtin-ls-files.c
+++ b/builtin-ls-files.c
@@ -10,6 +10,8 @@
 #include "dir.h"
 #include "builtin.h"
 #include "tree.h"
+#include "tree-walk.h"
+#include "unpack-trees.h"
 
 static int abbrev;
 static int show_deleted;
@@ -31,6 +33,7 @@ static const char **pathspec;
 static int error_unmatch;
 static char *ps_matched;
 static const char *with_tree;
+static struct narrow_spec *narrow_spec;
 
 static const char *tag_cached = "";
 static const char *tag_unmerged = "";
@@ -158,7 +161,8 @@ static void show_ce_entry(const char *tag, struct cache_entry *ce)
 	int len = prefix_len;
 	int offset = prefix_offset;
 
-	if (len >= ce_namelen(ce))
+	/* when narrow_spec->full_index is true, the_index is not pruned */
+	if (len >= ce_namelen(ce) && (!narrow_spec || !narrow_spec->full_index))
 		die("git ls-files: internal error - cache entry not superset of prefix");
 
 	if (pathspec && !pathspec_match(pathspec, ps_matched, ce->name, len))
@@ -231,6 +235,8 @@ static void show_files(struct dir_struct *dir, const char *prefix)
 			}
 			if (!(show_cached | show_stage))
 				continue;
+			if (narrow_spec && !match_narrow_spec(narrow_spec, ce->name))
+				continue;
 			show_ce_entry(ce_stage(ce) ? tag_unmerged : tag_cached, ce);
 		}
 	}
@@ -412,7 +418,7 @@ int report_path_error(const char *ps_matched, const char **pathspec, int prefix_
 
 static const char ls_files_usage[] =
 	"git ls-files [-z] [-t] [-v] (--[cached|deleted|others|stage|unmerged|killed|modified|orphaned|no-checkout])* "
-	"[ --sparse ] "
+	"[ --sparse ] [--narrow-match=<narrow_spec>] "
 	"[ --ignored ] [--exclude=<pattern>] [--exclude-from=<file>] "
 	"[ --exclude-per-directory=<filename> ] [--exclude-standard] "
 	"[--full-name] [--abbrev] [--] [<file>]*";
@@ -471,6 +477,10 @@ int cmd_ls_files(int argc, const char **argv, const char *prefix)
 			sparse_checkout = 1;
 			continue;
 		}
+		if (!prefixcmp(arg, "--narrow-match=")) {
+			narrow_spec = parse_narrow_spec(arg+15, prefix);
+			continue;
+		}
 		if (!strcmp(arg, "-d") || !strcmp(arg, "--deleted")) {
 			show_deleted = 1;
 			continue;
@@ -602,8 +612,14 @@ int cmd_ls_files(int argc, const char **argv, const char *prefix)
 	      show_killed | show_modified | show_orphaned | show_no_checkout))
 		show_cached = 1;
 
+	if (narrow_spec && !show_cached && !show_stage)
+		die("ls-files: --narrow-match can only be used with either --cached or --stage");
+
+	if (narrow_spec && narrow_spec->full_index && prefix_offset != 0)
+		die("ls-files: --narrow-match with root matching patterns requires --full-name");
+
 	read_cache();
-	if (prefix)
+	if (prefix && (!narrow_spec || !narrow_spec->full_index))
 		prune_cache(prefix);
 	if (with_tree) {
 		/*
@@ -625,5 +641,6 @@ int cmd_ls_files(int argc, const char **argv, const char *prefix)
 		return bad ? 1 : 0;
 	}
 
+	free_narrow_spec(narrow_spec);
 	return 0;
 }
diff --git a/t/t3003-ls-files-narrow-match.sh b/t/t3003-ls-files-narrow-match.sh
new file mode 100755
index 0000000..9879525
--- /dev/null
+++ b/t/t3003-ls-files-narrow-match.sh
@@ -0,0 +1,41 @@
+#!/bin/sh
+
+test_description='This test is for narrow spec matching'
+
+. test-lib.sh
+
+D="$(cd ..;pwd)"/t3003
+
+test_pattern() {
+	test_expect_success "pattern $1" '
+		(
+		if [ -n "'$3'" ]; then cd '$3'; fi
+		git ls-files --full-name --narrow-match="'"$2"'" > result &&
+		diff -u result "'"$D/$1"'"
+		)
+	'
+}
+
+test_expect_success 'setup' '
+	touch ./1 ./2 ./3 ./"1:2" &&
+	mkdir -p sub/subsub sub2 &&
+	touch sub/1 sub/2 sub/3 &&
+	touch sub2/1 sub2/2 sub2/3 &&
+	touch sub/subsub/1 sub/subsub/2 sub/subsub/3 &&
+	git add .
+'
+
+test_pattern 1 1
+test_pattern sub sub
+test_pattern sub-1 1 sub
+test_pattern root-sub-1 /1 sub
+test_pattern subsub-slash subsub/ sub
+test_pattern sub-only 'sub/:!sub/subsub/'
+test_pattern 12 1:2
+test_pattern cur-12 ./1:./2
+test_pattern slash-1 'sub/*1'
+test_pattern clone-escape '1\:2:1'
+test_pattern prefix-sub2 1:sub2//2 sub
+
+test_done
+
diff --git a/t/t3003/1 b/t/t3003/1
new file mode 100644
index 0000000..4c7381e
--- /dev/null
+++ b/t/t3003/1
@@ -0,0 +1,4 @@
+1
+sub/1
+sub/subsub/1
+sub2/1
diff --git a/t/t3003/12 b/t/t3003/12
new file mode 100644
index 0000000..0eeb4b8
--- /dev/null
+++ b/t/t3003/12
@@ -0,0 +1,8 @@
+1
+2
+sub/1
+sub/2
+sub/subsub/1
+sub/subsub/2
+sub2/1
+sub2/2
diff --git a/t/t3003/clone-escape b/t/t3003/clone-escape
new file mode 100644
index 0000000..31f866e
--- /dev/null
+++ b/t/t3003/clone-escape
@@ -0,0 +1,5 @@
+1
+1:2
+sub/1
+sub/subsub/1
+sub2/1
diff --git a/t/t3003/cur-12 b/t/t3003/cur-12
new file mode 100644
index 0000000..1191247
--- /dev/null
+++ b/t/t3003/cur-12
@@ -0,0 +1,2 @@
+1
+2
diff --git a/t/t3003/prefix-sub2 b/t/t3003/prefix-sub2
new file mode 100644
index 0000000..27c415c
--- /dev/null
+++ b/t/t3003/prefix-sub2
@@ -0,0 +1,3 @@
+sub/1
+sub/subsub/1
+sub2/2
diff --git a/t/t3003/root-sub-1 b/t/t3003/root-sub-1
new file mode 100644
index 0000000..d00491f
--- /dev/null
+++ b/t/t3003/root-sub-1
@@ -0,0 +1 @@
+1
diff --git a/t/t3003/slash-1 b/t/t3003/slash-1
new file mode 100644
index 0000000..5798e42
--- /dev/null
+++ b/t/t3003/slash-1
@@ -0,0 +1 @@
+sub/1
diff --git a/t/t3003/sub b/t/t3003/sub
new file mode 100644
index 0000000..e69de29
diff --git a/t/t3003/sub-1 b/t/t3003/sub-1
new file mode 100644
index 0000000..3ef951a
--- /dev/null
+++ b/t/t3003/sub-1
@@ -0,0 +1,2 @@
+sub/1
+sub/subsub/1
diff --git a/t/t3003/sub-only b/t/t3003/sub-only
new file mode 100644
index 0000000..3115212
--- /dev/null
+++ b/t/t3003/sub-only
@@ -0,0 +1,3 @@
+sub/1
+sub/2
+sub/3
diff --git a/t/t3003/subsub-slash b/t/t3003/subsub-slash
new file mode 100644
index 0000000..bc585b0
--- /dev/null
+++ b/t/t3003/subsub-slash
@@ -0,0 +1,3 @@
+sub/subsub/1
+sub/subsub/2
+sub/subsub/3
diff --git a/unpack-trees.c b/unpack-trees.c
index 54f301d..83888ae 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -726,6 +726,176 @@ static void show_stage_entry(FILE *o,
 }
 #endif
 
+struct narrow_spec *parse_narrow_spec(const char *spec, const char *prefix)
+{
+	struct narrow_spec *ns;
+	struct narrow_pattern *p;
+	const char *start, *end, *prefix_end;
+	int has_wildcards, has_slashes;
+
+	ns = xmalloc(sizeof(*ns));
+	memset(ns, 0, sizeof(*ns));
+	if (prefix)
+		ns->prefix = xstrdup(prefix);
+
+	start = spec;
+	while (*start) {
+		end = start;
+		prefix_end = NULL;
+		has_slashes = has_wildcards = 0;
+		while (*end && *end != ':') {
+			if (*end == '*' || *end == '[' || *end == '?')
+				has_wildcards = 1;
+			if (*end == '/') {
+				has_slashes = 1;
+
+				/* A prefix is terminated by double slashes */
+				if (end[1] == '/' && !prefix_end) {
+					prefix_end = end+2;
+					/* start over again */
+					has_slashes = has_wildcards = 0;
+				}
+			}
+			if (*end == '\\') {
+				end++;
+				has_wildcards = 1;
+				if (*end == '\0') /* trailing backslash */
+					break;
+			}
+			end++;
+		}
+		/* empty pattern or pattern with prefix only */
+		if (start == end || prefix_end == end)
+			goto cont;
+
+		p = xmalloc(sizeof(*p));
+		memset(p, 0, sizeof(*p));
+		if (prefix_end) {
+			int len = prefix_end - start - 1; /* only take one slash */
+			p->prefix = xmalloc(len + 1);
+			memcpy(p->prefix, start, len);
+			p->prefix[len] = '\0';
+			start = prefix_end;
+		}
+		if (*start == '!') {
+			p->negative = 1;
+			start++;
+		}
+		p->has_slashes = has_slashes;
+		p->has_wildcards = has_wildcards;
+		p->has_trailing_slash = end[-1] == '/';
+		if (*start == '/') {
+			p->has_root = 1;
+			start++;
+		}
+		else if (*start == '.' && start[1] == '/')
+			start += 2;
+		p->len = end-start;
+		p->pattern = xmalloc(p->len+1);
+		memcpy(p->pattern, start, p->len);
+		p->pattern[p->len] = '\0';
+
+		ALLOC_GROW(ns->patterns, ns->nr + 1, ns->alloc);
+		ns->patterns[ns->nr++] = p;
+
+		/*
+		 * ls-files needs to know whether it can prune the index
+		 * in these cases, index cannot be pruned
+		 */
+		if (p->has_root || p->prefix)
+			ns->full_index = 1;
+
+cont:
+		if (*end != ':')
+			break;
+		start = end + 1;
+	}
+	return ns;
+}
+
+void free_narrow_spec(struct narrow_spec *spec)
+{
+	int i;
+	if (!spec)
+		return;
+	for (i = 0; i < spec->nr; i++) {
+		struct narrow_pattern *p = spec->patterns[i];
+		if (p->prefix)
+			free(p->prefix);
+		if (p->pattern)
+			free(p->pattern);
+		free(p);
+	}
+	if (spec->patterns)
+		free(spec->patterns);
+	if (spec->prefix)
+		free(spec->prefix);
+	free(spec);
+}
+
+int match_narrow_spec(struct narrow_spec *spec, const char *path)
+{
+	int i, match;
+
+	if (!spec)
+		return 1; /* always match if spec is NULL */
+
+	for (i = spec->nr - 1;i >= 0; i--) {
+		struct narrow_pattern	*p	    = spec->patterns[i];
+		const char		*prefix	    = p->prefix   ? p->prefix      : spec->prefix;
+		int			 prefix_len = prefix      ? strlen(prefix) : 0;
+		const char		*new_path   = p->has_root ? path           : path + prefix_len;
+
+		if (!p->has_root && prefix && prefixcmp(path, prefix))
+			continue;
+
+		if (p->has_trailing_slash) {
+			/* the only "wildcard" here is backslash escape */
+			if (p->has_wildcards) {
+				char *unescaped_pattern = xstrdup(p->pattern);
+				char *src, *dst;
+
+				src = dst = unescaped_pattern;
+				while (*src) {
+					if (*src == '\\')
+						src++;
+					if (src != dst)
+						*dst = *src;
+					src++;
+					dst++;
+				}
+				*dst = '\0';
+				match = prefixcmp(new_path, unescaped_pattern) == 0;
+				free(unescaped_pattern);
+			}
+			else
+				match = prefixcmp(new_path, p->pattern) == 0;
+		}
+		else if (p->has_slashes) {
+			if (p->has_wildcards)
+				match = fnmatch(p->pattern, new_path, FNM_PATHNAME) == 0;
+			else
+				match = strcmp(p->pattern, new_path) == 0;
+		}
+		else {
+			const char *basename = strrchr(path + prefix_len, '/');
+			if (basename)
+				basename++;
+			else
+				basename = path + prefix_len;
+			if (p->has_wildcards)
+				match = fnmatch(p->pattern, basename, 0) == 0;
+			else
+				match = strcmp(p->pattern, basename) == 0;
+		}
+		if (match)
+			return p->negative ? 0 : 1;
+	}
+
+	/* no pattern is matched */
+	return 0;
+}
+
 int threeway_merge(struct cache_entry **stages, struct unpack_trees_options *o)
 {
 	struct cache_entry *index;
diff --git a/unpack-trees.h b/unpack-trees.h
index 0d26f3d..93ec3c6 100644
--- a/unpack-trees.h
+++ b/unpack-trees.h
@@ -16,6 +16,23 @@ struct unpack_trees_error_msgs {
 	const char *bind_overlap;
 };
 
+struct narrow_spec {
+	int nr;
+	int alloc;
+	int full_index;
+	char *prefix;
+	struct narrow_pattern {
+		int len;
+		int has_root:1;
+		int has_slashes:1;
+		int has_wildcards:1;
+		int has_trailing_slash:1;
+		int negative:1;
+		char *pattern;
+		char *prefix;
+	} **patterns;
+};
+
 struct unpack_trees_options {
 	unsigned int reset:1,
 		     merge:1,
@@ -48,6 +65,9 @@ struct unpack_trees_options {
 extern int unpack_trees(unsigned n, struct tree_desc *t,
 		struct unpack_trees_options *options);
 
+struct narrow_spec *parse_narrow_spec(const char *spec, const char *prefix);
+void free_narrow_spec(struct narrow_spec *spec);
+int match_narrow_spec(struct narrow_spec *spec, const char *path);
 int threeway_merge(struct cache_entry **stages, struct unpack_trees_options *o);
 int twoway_merge(struct cache_entry **src, struct unpack_trees_options *o);
 int bind_merge(struct cache_entry **src, struct unpack_trees_options *o);
-- 
1.6.0.3.890.g95457

[PATCH 3/8] unpack_trees(): keep track of unmerged entries

[Nguyễn Thái Ngọc Duy]

"git read-tree --reset" will currently remove all unmerged entries in
index before feeding the index to unpack_trees(). Because the lack of
unmerged entries, these entries, when read from tree, will be seen as
"new entries" by {one,two,three}way_merge().

This is fine for now. But for sparse checkout, it needs to know
whether an entry is new entry, because it will handle it different way
than already-in entry.

So the patch moves "unmerged entries removal" part into
unpack_trees(), actually unpack_callback(). The function then can turn
on o->has_unmerged flag, which can be utilized by sparse checkout.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin-read-tree.c |    3 ++-
 unpack-trees.c      |    7 +++++++
 unpack-trees.h      |    2 ++
 3 files changed, 11 insertions(+), 1 deletions(-)

diff --git a/builtin-read-tree.c b/builtin-read-tree.c
index 38fef34..528134c 100644
--- a/builtin-read-tree.c
+++ b/builtin-read-tree.c
@@ -136,9 +136,10 @@ int cmd_read_tree(int argc, const char **argv, const char *unused_prefix)
 			if (stage || opts.merge || opts.prefix)
 				usage(read_tree_usage);
 			opts.reset = 1;
+			opts.prune_unmerged = 1;
 			opts.merge = 1;
 			stage = 1;
-			read_cache_unmerged();
+			read_cache();
 			continue;
 		}
 
diff --git a/unpack-trees.c b/unpack-trees.c
index 83888ae..7d99051 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -301,6 +301,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str
 
 	/* Are we supposed to look at the index too? */
 	if (o->merge) {
+		o->has_unmerged = 0;
 		while (o->pos < o->src_index->cache_nr) {
 			struct cache_entry *ce = o->src_index->cache[o->pos];
 			int cmp = compare_entry(ce, info, p);
@@ -320,6 +321,12 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str
 						add_entry(o, ce, 0, 0);
 						return mask;
 					}
+
+					if (o->prune_unmerged) {
+						o->has_unmerged = 1;
+						/* leave src[0] as NULL and go over all other staged entries */
+						continue;
+					}
 				}
 				src[0] = ce;
 			}
diff --git a/unpack-trees.h b/unpack-trees.h
index 93ec3c6..86f0989 100644
--- a/unpack-trees.h
+++ b/unpack-trees.h
@@ -43,6 +43,8 @@ struct unpack_trees_options {
 		     verbose_update:1,
 		     aggressive:1,
 		     skip_unmerged:1,
+		     prune_unmerged:1,
+		     has_unmerged:1,
 		     initial_checkout:1,
 		     gently:1;
 	const char *prefix;
-- 
1.6.0.3.890.g95457

[PATCH 4/8] unpack_trees(): add support for sparse checkout

[Nguyễn Thái Ngọc Duy]

This patch teaches unpack_trees() to checkout/remove entries
on working directories appropriately when sparse checkout area is
changed. There are three kind of changes:

 - new_narrow_path: reset workdir to a completely new checkout area
 - add_narrow_path: keep current areas and add more entries
 - remove_narrow_path: remove some entries from current areas

When unpack_trees() is called without any of those options, changes in
no-checkout entries will not get checked out on working directory. New
files still appear on working directory though. That will be handled
later by core.defaultsparse

CE_WD_REMOVE is introduced to remove entries from working directories,
but still keep them in index

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 cache.h        |    3 +
 unpack-trees.c |  131 +++++++++++++++++++++++++++++++++++++++++++++++++++++---
 unpack-trees.h |    6 +++
 3 files changed, 134 insertions(+), 6 deletions(-)

diff --git a/cache.h b/cache.h
index 321fc54..b13df06 100644
--- a/cache.h
+++ b/cache.h
@@ -173,6 +173,9 @@ struct cache_entry {
 #define CE_HASHED    (0x100000)
 #define CE_UNHASHED  (0x200000)
 
+/* Only remove in work directory, not index */
+#define CE_WT_REMOVE (0x400000)
+
 /*
  * Extended on-disk flags
  */
diff --git a/unpack-trees.c b/unpack-trees.c
index 7d99051..a2794b8 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -31,6 +31,12 @@ static struct unpack_trees_error_msgs unpack_plumbing_errors = {
 
 	/* bind_overlap */
 	"Entry '%s' overlaps with '%s'.  Cannot bind.",
+
+	/* sparse_not_uptodate_file */
+	"Entry '%s' not uptodate. Cannot update sparse checkout.",
+
+	/* would_lose_orphaned */
+	"Orphaned working tree file '%s' would be %s by sparse checkout update.",
 };
 
 #define ERRORMSG(o,fld) \
@@ -96,7 +102,7 @@ static int check_updates(struct unpack_trees_options *o)
 	if (o->update && o->verbose_update) {
 		for (total = cnt = 0; cnt < index->cache_nr; cnt++) {
 			struct cache_entry *ce = index->cache[cnt];
-			if (ce->ce_flags & (CE_UPDATE | CE_REMOVE))
+			if (ce->ce_flags & (CE_UPDATE | CE_REMOVE | CE_WT_REMOVE))
 				total++;
 		}
 
@@ -108,6 +114,13 @@ static int check_updates(struct unpack_trees_options *o)
 	for (i = 0; i < index->cache_nr; i++) {
 		struct cache_entry *ce = index->cache[i];
 
+		if (ce->ce_flags & CE_WT_REMOVE) {
+			display_progress(progress, ++cnt);
+			if (o->update)
+				unlink_entry(ce);
+			continue;
+		}
+
 		if (ce->ce_flags & CE_REMOVE) {
 			display_progress(progress, ++cnt);
 			if (o->update)
@@ -133,6 +146,82 @@ static int check_updates(struct unpack_trees_options *o)
 	return errs != 0;
 }
 
+static int verify_uptodate_sparse(struct cache_entry *ce, struct unpack_trees_options *o);
+static int verify_absent_sparse(struct cache_entry *ce, const char *action, struct unpack_trees_options *o);
+static int apply_narrow_spec(struct unpack_trees_options *o)
+{
+	struct index_state *index = &o->result;
+	int i, len, namelen;
+	int header = 0;
+	char *special_files[] = { ".gitignore", GITATTRIBUTES_FILE, NULL };
+	char **special_file;
+
+	for (i = 0; i < index->cache_nr; i++) {
+		struct cache_entry *ce = index->cache[i];
+		int was_checkout = ce_checkout(ce);
+
+		if (ce_stage(ce))
+			continue;
+
+		/* Special case: .gitignore and .gitattributes should stay */
+		for (special_file = special_files; *special_file; special_file++) {
+			len = strlen(*special_file);
+			namelen = ce_namelen(ce);
+			if ((namelen == len ||
+			     (namelen > len && ce->name[namelen-len-1] == '/')) &&
+			    !strcmp(ce->name+namelen-len, *special_file))
+				break;
+		}
+		if (*special_file)
+			continue;
+
+		if (o->new_narrow_path | o->add_narrow_path | o->remove_narrow_path) {
+			int match = match_narrow_spec(o->narrow_spec, ce->name);
+			if (o->new_narrow_path) {
+				if (match)
+					ce_mark_checkout(ce);
+				else
+					ce_mark_no_checkout(ce);
+			}
+			if (o->add_narrow_path && match)
+				ce_mark_checkout(ce);
+			if (o->remove_narrow_path && match)
+				ce_mark_no_checkout(ce);
+		}
+
+		/* Update worktree, add/remove entries if needed */
+
+		/*
+		 * We only care about files getting into the checkout area
+		 * If merge strategies want to remove some, go ahead
+		 */
+		if (ce->ce_flags & CE_REMOVE)
+			continue;
+
+		if (was_checkout && ce_no_checkout(ce)) {
+			/*
+			 * If CE_UPDATE is set, verify_uptodate() must be called already
+			 * also stat info may have lost after merged_entry() so calling
+			 * verify_uptodate() again may fail
+			 */
+			if (!(ce->ce_flags & CE_UPDATE) && verify_uptodate_sparse(ce, o))
+				return -1;
+			ce->ce_flags |= CE_WT_REMOVE;
+		}
+		if (!was_checkout && ce_checkout(ce)) {
+			if (verify_absent_sparse(ce, "overwritten", o))
+				return -1;
+			ce->ce_flags |= CE_UPDATE;
+		}
+
+		/* merge strategies may set CE_UPDATE outside checkout area */
+		if (ce_no_checkout(ce))
+			ce->ce_flags &= ~CE_UPDATE;
+
+	}
+	return 0;
+}
+
 static inline int call_unpack_fn(struct cache_entry **src, struct unpack_trees_options *o)
 {
 	int ret = o->fn(src, o);
@@ -416,6 +505,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 	if (o->trivial_merges_only && o->nontrivial_merge)
 		return unpack_failed(o, "Merge requires file-level merging");
 
+	if (apply_narrow_spec(o))
+		return unpack_failed(o, NULL);
+
 	o->src_index = NULL;
 	ret = check_updates(o) ? (-2) : 0;
 	if (o->dst_index)
@@ -445,8 +537,9 @@ static int same(struct cache_entry *a, struct cache_entry *b)
  * When a CE gets turned into an unmerged entry, we
  * want it to be up-to-date
  */
-static int verify_uptodate(struct cache_entry *ce,
-		struct unpack_trees_options *o)
+static int verify_uptodate_generic(struct cache_entry *ce,
+				   struct unpack_trees_options *o,
+				   const char *error_msg)
 {
 	struct stat st;
 
@@ -471,7 +564,18 @@ static int verify_uptodate(struct cache_entry *ce,
 	if (errno == ENOENT)
 		return 0;
 	return o->gently ? -1 :
-		error(ERRORMSG(o, not_uptodate_file), ce->name);
+		error(error_msg, ce->name);
+}
+
+static int verify_uptodate(struct cache_entry *ce,
+			   struct unpack_trees_options *o)
+{
+	return verify_uptodate_generic(ce, o, ERRORMSG(o, not_uptodate_file));
+}
+static int verify_uptodate_sparse(struct cache_entry *ce,
+				  struct unpack_trees_options *o)
+{
+	return verify_uptodate_generic(ce, o, ERRORMSG(o, sparse_not_uptodate_file));
 }
 
 static void invalidate_ce_path(struct cache_entry *ce, struct unpack_trees_options *o)
@@ -583,8 +687,9 @@ static int icase_exists(struct unpack_trees_options *o, struct cache_entry *dst,
  * We do not want to remove or overwrite a working tree file that
  * is not tracked, unless it is ignored.
  */
-static int verify_absent(struct cache_entry *ce, const char *action,
-			 struct unpack_trees_options *o)
+static int verify_absent_generic(struct cache_entry *ce, const char *action,
+				 struct unpack_trees_options *o,
+				 const char *error_msg)
 {
 	struct stat st;
 
@@ -662,6 +767,16 @@ static int verify_absent(struct cache_entry *ce, const char *action,
 	}
 	return 0;
 }
+static int verify_absent(struct cache_entry *ce, const char *action,
+			 struct unpack_trees_options *o)
+{
+	return verify_absent_generic(ce, action, o, ERRORMSG(o, would_lose_untracked));
+}
+static int verify_absent_sparse(struct cache_entry *ce, const char *action,
+			 struct unpack_trees_options *o)
+{
+	return verify_absent_generic(ce, action, o, ERRORMSG(o, would_lose_orphaned));
+}
 
 static int merged_entry(struct cache_entry *merge, struct cache_entry *old,
 		struct unpack_trees_options *o)
@@ -684,6 +799,8 @@ static int merged_entry(struct cache_entry *merge, struct cache_entry *old,
 				return -1;
 			invalidate_ce_path(old, o);
 		}
+		if (ce_no_checkout(old))
+			update |= CE_NO_CHECKOUT;
 	}
 	else {
 		if (verify_absent(merge, "overwritten", o))
@@ -1182,6 +1299,8 @@ int oneway_merge(struct cache_entry **src, struct unpack_trees_options *o)
 			    ie_match_stat(o->src_index, old, &st, CE_MATCH_IGNORE_VALID))
 				update |= CE_UPDATE;
 		}
+		if (ce_no_checkout(old))
+			update |= CE_NO_CHECKOUT;
 		add_entry(o, old, update, 0);
 		return 0;
 	}
diff --git a/unpack-trees.h b/unpack-trees.h
index 86f0989..7e9febd 100644
--- a/unpack-trees.h
+++ b/unpack-trees.h
@@ -14,6 +14,8 @@ struct unpack_trees_error_msgs {
 	const char *not_uptodate_dir;
 	const char *would_lose_untracked;
 	const char *bind_overlap;
+	const char *sparse_not_uptodate_file;
+	const char *would_lose_orphaned;
 };
 
 struct narrow_spec {
@@ -46,6 +48,9 @@ struct unpack_trees_options {
 		     prune_unmerged:1,
 		     has_unmerged:1,
 		     initial_checkout:1,
+		     new_narrow_path:1,
+		     add_narrow_path:2,
+		     remove_narrow_path:2,
 		     gently:1;
 	const char *prefix;
 	int pos;
@@ -57,6 +62,7 @@ struct unpack_trees_options {
 	int merge_size;
 
 	struct cache_entry *df_conflict_entry;
+	struct narrow_spec *narrow_spec;
 	void *unpack_data;
 
 	struct index_state *dst_index;
-- 
1.6.0.3.890.g95457

[PATCH 5/8] clone: support sparse checkout with --sparse-checkout option

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-clone.txt |   10 +++++++++-
 builtin-clone.c             |   14 ++++++++++++++
 t/t5703-clone-narrow.sh     |   41 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 64 insertions(+), 1 deletions(-)
 create mode 100755 t/t5703-clone-narrow.sh

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 95f08b9..acdced2 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -12,7 +12,8 @@ SYNOPSIS
 'git clone' [--template=<template_directory>]
 	  [-l] [-s] [--no-hardlinks] [-q] [-n] [--bare] [--mirror]
 	  [-o <name>] [-u <upload-pack>] [--reference <repository>]
-	  [--depth <depth>] [--] <repository> [<directory>]
+	  [--depth <depth>] [-S|--sparse-checkout=<sparse patterns>] [--]
+	  <repository> [<directory>]
 
 DESCRIPTION
 -----------
@@ -99,6 +100,13 @@ then the cloned repository will become corrupt.
 -n::
 	No checkout of HEAD is performed after the clone is complete.
 
+-S=<sparse patterns>::
+--sparse-checkout=<sparse patterns>::
+	Make a sparse checkout instead of a full one.
+	This option will not work with either --no-checkout or --bare.
+	Please refer to linkgit:git-checkout[1] for more detail on
+	sparse checkout and sparse patterns.
+
 --bare::
 	Make a 'bare' GIT repository.  That is, instead of
 	creating `<directory>` and placing the administrative
diff --git a/builtin-clone.c b/builtin-clone.c
index 2feac9c..ea341a1 100644
--- a/builtin-clone.c
+++ b/builtin-clone.c
@@ -36,6 +36,7 @@ static const char * const builtin_clone_usage[] = {
 static int option_quiet, option_no_checkout, option_bare, option_mirror;
 static int option_local, option_no_hardlinks, option_shared;
 static char *option_template, *option_reference, *option_depth;
+static char *option_narrow_path;
 static char *option_origin = NULL;
 static char *option_upload_pack = "git-upload-pack";
 static int option_verbose;
@@ -45,6 +46,8 @@ static struct option builtin_clone_options[] = {
 	OPT__VERBOSE(&option_verbose),
 	OPT_BOOLEAN('n', "no-checkout", &option_no_checkout,
 		    "don't create a checkout"),
+	OPT_STRING('S', "sparse-checkout", &option_narrow_path, "sparse patterns",
+		   "only checkout certain files based on patterns"),
 	OPT_BOOLEAN(0, "bare", &option_bare, "create a bare repository"),
 	OPT_BOOLEAN(0, "naked", &option_bare, "create a bare repository"),
 	OPT_BOOLEAN(0, "mirror", &option_mirror,
@@ -383,10 +386,15 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 		if (option_origin)
 			die("--bare and --origin %s options are incompatible.",
 			    option_origin);
+		if (option_narrow_path)
+			die("--bare and --sparse-checkout options are incompatible.");
 		option_no_checkout = 1;
 		use_separate_remote = 0;
 	}
 
+	if (option_no_checkout && option_narrow_path)
+		die("--no-checkout and --sparse-checkout options are incompatible.");
+
 	if (!option_origin)
 		option_origin = "origin";
 
@@ -597,10 +605,16 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 		opts.src_index = &the_index;
 		opts.dst_index = &the_index;
 
+		if (option_narrow_path) {
+			opts.new_narrow_path = 1;
+			opts.narrow_spec = parse_narrow_spec(option_narrow_path, NULL);
+		}
+
 		tree = parse_tree_indirect(remote_head->old_sha1);
 		parse_tree(tree);
 		init_tree_desc(&t, tree->buffer, tree->size);
 		unpack_trees(1, &t, &opts);
+		free_narrow_spec(opts.narrow_spec);
 
 		if (write_cache(fd, active_cache, active_nr) ||
 		    commit_locked_index(lock_file))
diff --git a/t/t5703-clone-narrow.sh b/t/t5703-clone-narrow.sh
new file mode 100755
index 0000000..e34246d
--- /dev/null
+++ b/t/t5703-clone-narrow.sh
@@ -0,0 +1,41 @@
+#!/bin/sh
+
+test_description='narrow clone'
+
+. ./test-lib.sh
+
+test_expect_success setup '
+	rm -fr .git &&
+	test_create_repo src &&
+	(
+		cd src
+		mkdir -p work/sub/dir
+		touch untracked tracked modified added
+		touch work/untracked work/tracked work/modified work/added
+		git add tracked work/tracked
+		git add modified work/modified
+		git commit -m initial
+	)
+
+'
+
+test_expect_success 'narrow clone incompatible with --bare' '
+	rm -fr dst &&
+	test_must_fail git clone --sparse-checkout=work/ --bare src dst
+'
+
+test_expect_success 'narrow clone incompatible with --no-checkout' '
+	rm -fr dst &&
+	test_must_fail git clone --sparse-checkout=work/ -n src dst
+'
+
+test_expect_success 'clone with --sparse-checkout' '
+	(
+	rm -fr dst &&
+	git clone --sparse-checkout=work/ src dst &&
+	cd dst &&
+	test -z "$(git ls-files --sparse | grep -v ^work/)"
+	)
+'
+
+test_done
-- 
1.6.0.3.890.g95457

[PATCH 6/8] checkout: add new options to support sparse checkout

[Nguyễn Thái Ngọc Duy]

This patch adds main interface to manipulate sparse checkout.
New options are added to support entering/updating/leaving sparse
checkout:

 --full: return to full checkout (default)
 --sparse: set checkout area according to given spec
 --{include,exclude}-sparse: adjust current sparse checkout area

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-checkout.txt |   55 ++++++++++++++++-
 builtin-checkout.c             |   39 ++++++++++++
 t/t2011-checkout-sparse.sh     |  128 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 219 insertions(+), 3 deletions(-)
 create mode 100755 t/t2011-checkout-sparse.sh

diff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt
index 0813d9f..b32043f 100644
--- a/Documentation/git-checkout.txt
+++ b/Documentation/git-checkout.txt
@@ -8,8 +8,10 @@ git-checkout - Checkout a branch or paths to the working tree
 SYNOPSIS
 --------
 [verse]
-'git checkout' [-q] [-f] [--track | --no-track] [-b <new_branch> [-l]] [-m] [<branch>]
-'git checkout' [-f|--ours|--theirs|-m|--conflict=<style>] [<tree-ish>] [--] <paths>...
+'git checkout' [-q] [-f] [--track | --no-track] [-b <new_branch> [-l]] [-m]
+	  [<sparse checkout options>] [<branch>]
+'git checkout' [-f|--ours|--theirs|-m|--conflict=<style>] [<tree-ish>]
+	  [<sparse checkout options>] [--] <paths>...
 
 DESCRIPTION
 -----------
@@ -34,6 +36,10 @@ used to specify a specific tree-ish (i.e. commit, tag or tree)
 to update the index for the given paths before updating the
 working tree.
 
+<sparse checkout options> include --full, --sparse, --include-sparse
+and --exclude-sparse. The last three require sparse patterns. Please refer
+to "sparse checkout" section for more information about this mode.
+
 The index may contain unmerged entries after a failed merge.  By
 default, if you try to check out such an entry from the index, the
 checkout operation will fail and nothing will be checked out.
@@ -117,6 +123,35 @@ should result in deletion of the path).
 When checking out paths from the index, this option lets you recreate
 the conflicted merge in the specified paths.
 
+--full::
+	Quit sparse checkout mode. Return to full checkout.
+	This option cannot be used with either --sparse,
+	--include-sparse, --exclude-sparse or <paths>.
+
+-S=<sparse patterns>::
+--sparse=<sparse patterns>::
+	Re-apply new sparse patterns on current working directory to
+	form new checkout area. All no-checkout bits will be wiped
+	out before applying the patterns.
+	This option cannot be used with --full, --include-sparse,
+	--exclude-sparse or <paths>. Multiple --sparse is not allowed.
+
+-I=<sparse patterns>::
+--include-sparse=<sparse patterns>::
+	Checkout more areas specified by sparse patterns to current
+	checkout area. Already checked out entries are not affected.
+	This option cannot be used with --full, --sparse,
+	--exclude-sparse or <paths>. Multiple --include-sparse is not allowed.
+
+-E=<sparse patterns>::
+--exclude-sparse=<sparse patterns>::
+	Narrow checkout area by removing files specified by sparse patterns
+	from current checkout area. This operation will fail if there
+	are unmerged or modified files in the removing areas. No-checkout
+	entries are not affected.
+	This option cannot be used with --full, --sparse,
+	--include-sparse or <paths>. Multiple --exclude-sparse is not allowed.
+
 --conflict=<style>::
 	The same as --merge option above, but changes the way the
 	conflicting hunks are presented, overriding the
@@ -186,7 +221,10 @@ Because sparse checkout uses a new index format, it will be
 incompatible with git prior to 1.6.0 regarding worktree operations.
 Operations that only need access to the repository itself, such as
 clone, push, or pull/fetch from another (normal) repository... should
-not be affected by sparse checkout.
+not be affected by sparse checkout. In order to make your working
+directory work again with those versions, you can use
+`git checkout --full` to return to normal mode (and compatible index
+format).
 
 In sparse checkout mode, checkout status of every files in your
 working directory will be recorded in index. If a file is marked
@@ -263,6 +301,17 @@ Patterns have the following format:
  - Sparse patterns do not apply to .gitignore and .gitattributes
    files. They are always checked out.
 
+When you apply new sparse patterns to your working directory using either
+--sparse, --include-sparse or --exclude-sparse, it will update "checkout" status
+in index accordingly. Moreover, if a file is marked "no-checkout" and
+is present in working directory, it will be removed. If a file is
+turned from "no-checkout" to "checkout", then it will be added again
+to working directory. Unmerged entries will always be "checkout" regardless
+the sparse patterns. Modified entries will refuse to become "no-checkout".
+
+You can form your checkout area in one go with --sparse option,
+or do it incrementally with --include-sparse and --exclude-sparse.
+
 EXAMPLES
 --------
 
diff --git a/builtin-checkout.c b/builtin-checkout.c
index 7f3bd7b..be4cd3a 100644
--- a/builtin-checkout.c
+++ b/builtin-checkout.c
@@ -33,6 +33,12 @@ struct checkout_opts {
 	const char *new_branch;
 	int new_branch_log;
 	enum branch_track track;
+
+	const char *prefix;
+	char *new_path;
+	char *add_path;
+	char *remove_path;
+	int all_path;
 };
 
 static int post_checkout_hook(struct commit *old, struct commit *new,
@@ -388,6 +394,7 @@ static int merge_working_tree(struct checkout_opts *opts,
 		topts.dst_index = &the_index;
 
 		topts.msgs.not_uptodate_file = "You have local changes to '%s'; cannot switch branches.";
+		topts.msgs.sparse_not_uptodate_file = "You have local changes to '%s'; cannot update sparse checkout.";
 
 		refresh_cache(REFRESH_QUIET);
 
@@ -411,7 +418,25 @@ static int merge_working_tree(struct checkout_opts *opts,
 		tree = parse_tree_indirect(new->commit->object.sha1);
 		init_tree_desc(&trees[1], tree->buffer, tree->size);
 
+		if (opts->all_path) {
+			/* leave narrow_spec NULL */
+			topts.new_narrow_path = 1;
+		}
+		else if (opts->new_path) {
+			topts.narrow_spec = parse_narrow_spec(opts->new_path, opts->prefix);
+			topts.new_narrow_path = 1;
+		}
+		else if (opts->add_path) {
+			topts.narrow_spec = parse_narrow_spec(opts->add_path, opts->prefix);
+			topts.add_narrow_path = 1;
+		}
+		else if (opts->remove_path) {
+			topts.narrow_spec = parse_narrow_spec(opts->remove_path, opts->prefix);
+			topts.remove_narrow_path = 1;
+		}
+
 		ret = unpack_trees(2, trees, &topts);
+		free_narrow_spec(topts.narrow_spec);
 		if (ret == -1) {
 			/*
 			 * Unpack couldn't do a trivial merge; either
@@ -598,6 +623,10 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
 		OPT_BOOLEAN('m', "merge", &opts.merge, "merge"),
 		OPT_STRING(0, "conflict", &conflict_style, "style",
 			   "conflict style (merge or diff3)"),
+		OPT_BOOLEAN(0, "full", &opts.all_path, "quit sparse checkout"),
+		OPT_STRING('S', "sparse", &opts.new_path, "sparse patterns", "set new sparse checkout"),
+		OPT_STRING('I', "include-sparse", &opts.add_path, "sparse patterns", "widen checkout area"),
+		OPT_STRING('E', "exclude-sparse", &opts.remove_path, "sparse patterns", "narrow checkout area"),
 		OPT_END(),
 	};
 	int has_dash_dash;
@@ -608,6 +637,7 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
 	git_config(git_checkout_config, NULL);
 
 	opts.track = BRANCH_TRACK_UNSPECIFIED;
+	opts.prefix = prefix;
 
 	argc = parse_options(argc, argv, options, checkout_usage,
 			     PARSE_OPT_KEEP_DASHDASH);
@@ -634,6 +664,12 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
 		git_xmerge_config("merge.conflictstyle", conflict_style, NULL);
 	}
 
+	if (((opts.all_path ? 1 : 0) +
+	     (opts.new_path ? 1 : 0) +
+	     (opts.add_path ? 1 : 0) +
+	     (opts.remove_path ? 1 : 0)) > 1)
+		die("git checkout: --full, --sparse, --include-sparse and --exclude-sparse are incompatible");
+
 	if (opts.force && opts.merge)
 		die("git checkout: -f and -m are incompatible");
 
@@ -727,6 +763,9 @@ no_reference:
 		if (1 < !!opts.writeout_stage + !!opts.force + !!opts.merge)
 			die("git checkout: --ours/--theirs, --force and --merge are incompatible when\nchecking out of the index.");
 
+		if (opts.all_path || opts.new_path || opts.add_path || opts.remove_path)
+			die("git checkout: updating paths is incompatible with setting sparse checkout");
+
 		return checkout_paths(source_tree, pathspec, &opts);
 	}
 
diff --git a/t/t2011-checkout-sparse.sh b/t/t2011-checkout-sparse.sh
new file mode 100755
index 0000000..67aea96
--- /dev/null
+++ b/t/t2011-checkout-sparse.sh
@@ -0,0 +1,128 @@
+#!/bin/sh
+
+test_description='sparse checkout'
+
+. ./test-lib.sh
+
+test_expect_success setup '
+	mkdir work1 work2 work3
+	touch one two three
+	touch work1/one work2/two work3/three
+	touch .gitignore .gitattributes
+	touch work3/.gitignore work3/.gitattributes
+	git add .gitignore .gitattributes
+	git add work3/.gitignore work3/.gitattributes
+	git add one work1/one
+	git commit -m work1 && WORK1=$(git rev-parse HEAD)
+	echo conflict >> work1/one
+	git add work1/one
+	git commit -m "work1 conflict" && WORK1CONFLICT=$(git rev-parse HEAD)
+	git checkout $WORK1
+	echo one >> one
+	echo work1/1 >> work1/one
+	git add one work1/one
+	git commit -m "work1 modified" && WORK1MODIFIED=$(git rev-parse HEAD)
+	git add two work2/two
+	git commit -m work2 && WORK2=$(git rev-parse HEAD)
+	git add three work3/three
+	git commit -m work3 && WORK3=$(git rev-parse HEAD)
+'
+
+test_expect_success '--full on no-narrow checkout' '
+	git checkout --full
+'
+
+test_expect_success '--full and --sparse incompatible' '
+	test_must_fail git checkout --full --sparse=work1
+'
+
+test_expect_success 'limit worktree to work1 and work2' '
+	git checkout --sparse=work1/:work2/ &&
+	test -f work1/one &&
+	test -f work2/two &&
+	! test -f work3/three
+'
+
+test_expect_success 'exit sparse checkout' '
+	git checkout --full &&
+	test -f work1/one &&
+	test -f work2/two &&
+	test -f work3/three &&
+	test -f one
+'
+
+test_expect_success 'sparse checkout does not touch .git*' '
+	git checkout --sparse=work1/:work2/ &&
+	test -f .gitignore &&
+	test -f .gitattributes &&
+	test -f work3/.gitignore &&
+	test -f work3/.gitattributes
+'
+
+# merged_entry case with old != NULL, same_entry = 1
+# work1/ turns to no-checkout
+# work3/ turns to checkout
+test_expect_success 'update worktree to work2 and work3' '
+	git checkout --sparse=work2/:work3/ &&
+	! test -f work1/one &&
+	test -f work2/two &&
+	test -f work3/three
+'
+
+test_expect_success 'update narrow prefix with modification' '
+	echo modified >> work2/two &&
+	git checkout --sparse=work1/:work2/ &&
+	test -f work1/one &&
+	test -f work2/two &&
+	! test -f work3/three &&
+	grep -q modified work2/two &&
+
+	! git checkout --sparse=work1/:work3/ &&
+	test -f work1/one &&
+	test -f work2/two &&
+	! test -f work3/three &&
+	grep -q modified work2/two &&
+	git checkout work2/two
+'
+
+test_expect_success 'widen checkout area' '
+	git checkout --include-sparse=work3/ &&
+	test -f work1/one &&
+	test -f work2/two &&
+	test -f work3/three
+'
+
+test_expect_success 'narrow checkout area' '
+	git checkout --exclude-sparse=work3/ &&
+	test -f work1/one &&
+	test -f work2/two &&
+	! test -f work3/three
+'
+
+test_expect_success 'update outside checkout area' '
+	git checkout --full &&
+	git checkout $WORK1 &&
+	git checkout --sparse work2/ &&
+	! test -f work1/one &&
+	git checkout $WORK1MODIFIED &&
+	! test -f work1/one &&
+	git checkout --full master
+'
+
+test_expect_success 'conflict outside checkout area' '
+	git checkout --sparse="work2/" $WORK1MODIFIED &&
+	test -z "$(git ls-files --sparse work1/one)" &&
+	git merge $WORK1CONFLICT
+	test $? = 1 &&
+	test -n "$(git ls-files --sparse work1/one)" &&
+	git reset --hard &&
+	git checkout master
+'
+
+test_expect_success 'removal outside checkout area' '
+	git rm work1/one &&
+	git commit -m remove &&
+	git checkout --sparse=work2/ HEAD^
+'
+
+test_done
-- 
1.6.0.3.890.g95457

[PATCH 7/8] Introduce default sparse patterns (core.defaultsparse)

[Nguyễn Thái Ngọc Duy]

As noted in the previous commit message, new files will always get added into
working directory by unpack_trees(). This can be annoying so we keep
track of the sparse patterns people use to form their checkout area
and apply the patterns on new files. Those files that get filtered out
by this will be reported if verbose_update is TRUE.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/config.txt       |   10 +++++++
 Documentation/git-checkout.txt |   21 ++++++++++++++-
 Documentation/git-clone.txt    |    1 +
 builtin-checkout.c             |   54 ++++++++++++++++++++++++++++++++++++++++
 builtin-clone.c                |    2 +
 cache.h                        |    2 +
 config.c                       |    5 +++
 environment.c                  |    1 +
 t/t2011-checkout-sparse.sh     |   19 ++++++++++++++
 t/t5703-clone-narrow.sh        |    1 +
 unpack-trees.c                 |   52 ++++++++++++++++++++++++++++++++++++++
 unpack-trees.h                 |    1 +
 12 files changed, 167 insertions(+), 2 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index b233fe5..2b2dad1 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -422,6 +422,16 @@ relatively high IO latencies.  With this set to 'true', git will do the
 index comparison to the filesystem data in parallel, allowing
 overlapping IO's.
 
+core.defaultsparse::
+	The default sparse patterns that will be used to decide
+	whether a new file should be added into working
+	directory. If a new file matches the default sparse
+	patterns, then it will be checked out. Otherwise it will
+	be ignored and marked "no-checkout" in index.
+	Sparse update operations via "git clone" or "git checkout" will
+	automatically update this config. See linkgit:git-clone[1] and
+	linkgit:git-checkout[1] for more information.
+
 alias.*::
 	Command aliases for the linkgit:git[1] command wrapper - e.g.
 	after defining "alias.last = cat-file commit HEAD", the invocation
diff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt
index b32043f..892a4bb 100644
--- a/Documentation/git-checkout.txt
+++ b/Documentation/git-checkout.txt
@@ -125,6 +125,7 @@ the conflicted merge in the specified paths.
 
 --full::
 	Quit sparse checkout mode. Return to full checkout.
+	`core.defaultsparse` will be removed.
 	This option cannot be used with either --sparse,
 	--include-sparse, --exclude-sparse or <paths>.
 
@@ -133,6 +134,7 @@ the conflicted merge in the specified paths.
 	Re-apply new sparse patterns on current working directory to
 	form new checkout area. All no-checkout bits will be wiped
 	out before applying the patterns.
+	`core.defaultsparse` will be set with the given patterns.
 	This option cannot be used with --full, --include-sparse,
 	--exclude-sparse or <paths>. Multiple --sparse is not allowed.
 
@@ -140,6 +142,11 @@ the conflicted merge in the specified paths.
 --include-sparse=<sparse patterns>::
 	Checkout more areas specified by sparse patterns to current
 	checkout area. Already checked out entries are not affected.
+	`core.defaultsparse` will be appended with the given patterns.
+	Repetitive use of --include-sparse and --exclude-sparse may
+	leave cruft in `core.defaultsparse`, and may slow down
+	checkout operations. You may want to clean up
+	`core.defaultsparse` with --sparse.
 	This option cannot be used with --full, --sparse,
 	--exclude-sparse or <paths>. Multiple --include-sparse is not allowed.
 
@@ -149,6 +156,11 @@ the conflicted merge in the specified paths.
 	from current checkout area. This operation will fail if there
 	are unmerged or modified files in the removing areas. No-checkout
 	entries are not affected.
+	`core.defaultsparse` will be appended with the negated version of the given patterns.
+	Repetitive use of --include-sparse and --exclude-sparse may
+	leave cruft in `core.defaultsparse`, and may slow down
+	checkout operations. You may want to clean up
+	`core.defaultsparse` with --sparse.
 	This option cannot be used with --full, --sparse,
 	--include-sparse or <paths>. Multiple --exclude-sparse is not allowed.
 
@@ -230,8 +242,11 @@ In sparse checkout mode, checkout status of every files in your
 working directory will be recorded in index. If a file is marked
 "no-checkout", it means that file is not needed to be present in
 working directory by user or any git command. When a new file is added
-to index, it will be marked "checkout" unless sparse patterns are
-applied.  Unmerged files are always "checkout". When you checkout new
+to index, it will be checked with default sparse pattern (see
+`core.defaultsparse` in linkgit:git-config.txt[1]). If the file
+matches the pattern, it will be marked "checkout" and added to
+working directory. Otherwise it will be marked "no-checkout" in
+index.  Unmerged files are always "checkout". When you checkout new
 files using "git checkout <file>" they will be automatically marked
 "checkout". Other commands such as "git apply" can also checkout new
 files if they are needed. linkgit:git-update-index[1] can be used to
@@ -311,6 +326,8 @@ the sparse patterns. Modified entries will refuse to become "no-checkout".
 
 You can form your checkout area in one go with --sparse option,
 or do it incrementally with --include-sparse and --exclude-sparse.
+Patterns used in those options will be automatically recorded in
+`core.defaultsparse`.
 
 EXAMPLES
 --------
diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index acdced2..f7ccf15 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -103,6 +103,7 @@ then the cloned repository will become corrupt.
 -S=<sparse patterns>::
 --sparse-checkout=<sparse patterns>::
 	Make a sparse checkout instead of a full one.
+	`core.defaultsparse` will be set with the given sparse patterns.
 	This option will not work with either --no-checkout or --bare.
 	Please refer to linkgit:git-checkout[1] for more detail on
 	sparse checkout and sparse patterns.
diff --git a/builtin-checkout.c b/builtin-checkout.c
index be4cd3a..8c6d5c5 100644
--- a/builtin-checkout.c
+++ b/builtin-checkout.c
@@ -592,6 +592,60 @@ static int switch_branches(struct checkout_opts *opts, struct branch_info *new)
 
 	update_refs_for_switch(opts, &old, new);
 
+	/* Update core.defaultsparse */
+	if (opts->all_path ||
+	    (opts->new_path && !opts->prefix)) /* optimization when prefix is NULL */
+		git_config_set("core.defaultsparse", opts->new_path);
+	else if (opts->new_path || opts->add_path || opts->remove_path) {
+		const char *narrow_spec;
+		struct narrow_spec *spec;
+		struct strbuf sb;
+		int i;
+
+		if (opts->new_path)
+			narrow_spec = opts->new_path;
+		else if (opts->add_path)
+			narrow_spec = opts->add_path;
+		else
+			narrow_spec = opts->remove_path;
+		spec = parse_narrow_spec(narrow_spec, NULL) ;
+		strbuf_init(&sb, 0);
+
+		/*
+		 * --sparse does not save old core.defaultsparse
+		 * --include-sparse and --exclude-sparse on the other hand
+		 * append to the old core.defaultsparse
+		 */
+		if (core_default_sparse && !opts->new_path) {
+			strbuf_addstr(&sb, core_default_sparse);
+			strbuf_addch(&sb, ':');
+		}
+
+		/*
+		 * This happens when we do full checkout,
+		 * then --exclude-sparse.
+		 * Add full pattern first so the next
+		 *  pattern has something to negate
+		 */
+		if (!core_default_sparse && opts->remove_path)
+			strbuf_addstr(&sb, "/:");
+
+		for (i = 0; i < spec->nr; i++) {
+			struct narrow_pattern *p = spec->patterns[i];
+
+			if (i)
+				strbuf_addch(&sb, ':');
+			if (!p->prefix && opts->prefix)
+				p->prefix = xstrdup(opts->prefix);
+			if (opts->remove_path)
+				p->negative = !p->negative;
+			narrow_spec_to_strbuf(p, &sb);
+		}
+		git_config_set("core.defaultsparse", sb.buf);
+		free_narrow_spec(spec);
+		strbuf_release(&sb);
+	}
+
 	ret = post_checkout_hook(old.commit, new->commit, 1);
 	return ret || opts->writeout_error;
 }
diff --git a/builtin-clone.c b/builtin-clone.c
index ea341a1..209030a 100644
--- a/builtin-clone.c
+++ b/builtin-clone.c
@@ -619,6 +619,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 		if (write_cache(fd, active_cache, active_nr) ||
 		    commit_locked_index(lock_file))
 			die("unable to write new index file");
+
+		git_config_set("core.defaultsparse", option_narrow_path);
 	}
 
 	strbuf_release(&reflog_msg);
diff --git a/cache.h b/cache.h
index b13df06..a3366da 100644
--- a/cache.h
+++ b/cache.h
@@ -175,6 +175,7 @@ struct cache_entry {
 
 /* Only remove in work directory, not index */
 #define CE_WT_REMOVE (0x400000)
+#define CE_NEW       (0x800000)
 
 /*
  * Extended on-disk flags
@@ -522,6 +523,7 @@ extern size_t delta_base_cache_limit;
 extern int auto_crlf;
 extern int fsync_object_files;
 extern int core_preload_index;
+extern const char *core_default_sparse;
 
 enum safe_crlf {
 	SAFE_CRLF_FALSE = 0,
diff --git a/config.c b/config.c
index d2fc8f5..ca2cf03 100644
--- a/config.c
+++ b/config.c
@@ -495,6 +495,11 @@ static int git_default_core_config(const char *var, const char *value)
 		return 0;
 	}
 
+	if (!strcmp(var, "core.defaultsparse")) {
+		core_default_sparse = xstrdup(value);
+		return 0;
+	}
+
 	/* Add other config variables here and to Documentation/config.txt. */
 	return 0;
 }
diff --git a/environment.c b/environment.c
index e278bce..5a66e30 100644
--- a/environment.c
+++ b/environment.c
@@ -30,6 +30,7 @@ int zlib_compression_level = Z_BEST_SPEED;
 int core_compression_level;
 int core_compression_seen;
 int fsync_object_files;
+const char *core_default_sparse;
 size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
 size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
 size_t delta_base_cache_limit = 16 * 1024 * 1024;
diff --git a/t/t2011-checkout-sparse.sh b/t/t2011-checkout-sparse.sh
index 67aea96..15f3761 100755
--- a/t/t2011-checkout-sparse.sh
+++ b/t/t2011-checkout-sparse.sh
@@ -45,6 +45,7 @@ test_expect_success 'limit worktree to work1 and work2' '
 
 test_expect_success 'exit sparse checkout' '
 	git checkout --full &&
+	test -z "$(git config core.defaultsparse)" &&
 	test -f work1/one &&
 	test -f work2/two &&
 	test -f work3/three &&
@@ -125,4 +126,22 @@ test_expect_success 'removal outside checkout area' '
 	git checkout --sparse=work2/ HEAD^
 '
 
+test_expect_success 'core.defaultsparse, new files get filtered' '
+	git checkout --sparse=work2/ $WORK2 &&
+	git config core.defaultsparse "./*:work2/"
+	git checkout $WORK3 &&
+	test -f three &&
+	! test -f work3/three &&
+	git config --unset core.defaultsparse
+'
+
+test_expect_success 'save core.defaultsparse' '
+	git checkout --sparse="./*:work2/" $WORK2 &&
+	test "$(git config core.defaultsparse)" = "./*:work2/" &&
+	cd work3 && git checkout --include-sparse=1 $WORK2 && cd .. &&
+	test "$(git config core.defaultsparse)" = "./*:work2/:work3//1" &&
+	git checkout --exclude-sparse=work3/ $WORK2 &&
+	test "$(git config core.defaultsparse)" = "./*:work2/:work3//1:!work3/"
+'
+
 test_done
diff --git a/t/t5703-clone-narrow.sh b/t/t5703-clone-narrow.sh
index e34246d..691bb40 100755
--- a/t/t5703-clone-narrow.sh
+++ b/t/t5703-clone-narrow.sh
@@ -34,6 +34,7 @@ test_expect_success 'clone with --sparse-checkout' '
 	rm -fr dst &&
 	git clone --sparse-checkout=work/ src dst &&
 	cd dst &&
+	test "$(git config core.defaultsparse)" = "work/" &&
 	test -z "$(git ls-files --sparse | grep -v ^work/)"
 	)
 '
diff --git a/unpack-trees.c b/unpack-trees.c
index a2794b8..ad26987 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -151,6 +151,7 @@ static int verify_absent_sparse(struct cache_entry *ce, const char *action, stru
 static int apply_narrow_spec(struct unpack_trees_options *o)
 {
 	struct index_state *index = &o->result;
+	static struct narrow_spec *default_spec = NULL;
 	int i, len, namelen;
 	int header = 0;
 	char *special_files[] = { ".gitignore", GITATTRIBUTES_FILE, NULL };
@@ -188,6 +189,25 @@ static int apply_narrow_spec(struct unpack_trees_options *o)
 			if (o->remove_narrow_path && match)
 				ce_mark_no_checkout(ce);
 		}
+		/*
+		 * Default spec only applies if no directive is given
+		 * and it only applies to new files
+		 */
+		else if (ce->ce_flags & CE_NEW) {
+			if (!default_spec && core_default_sparse)
+				default_spec = parse_narrow_spec(core_default_sparse, NULL);
+
+			if (default_spec && !match_narrow_spec(default_spec, ce->name)) {
+				if (o->update && o->verbose_update) {
+					if (!header) {
+						fprintf(stderr, "New files get filtered out by default sparse patterns:\n");
+						header = 1;
+					}
+					fprintf(stderr, " %s\n", ce->name);
+				}
+				ce_mark_no_checkout(ce);
+			}
+		}
 
 		/* Update worktree, add/remove entries if needed */
 
@@ -805,6 +825,8 @@ static int merged_entry(struct cache_entry *merge, struct cache_entry *old,
 	else {
 		if (verify_absent(merge, "overwritten", o))
 			return -1;
+		if (!o->has_unmerged)
+			update |= CE_NEW;
 		invalidate_ce_path(merge, o);
 	}
 
@@ -957,6 +979,36 @@ void free_narrow_spec(struct narrow_spec *spec)
 	free(spec);
 }
 
+void narrow_spec_to_strbuf(struct narrow_pattern *p, struct strbuf *sb)
+{
+	const char *s;
+	if (p->prefix && !p->has_root) {
+		s = p->prefix;
+		while (*s) {
+			if (*s == ':')
+				strbuf_addch(sb, '\\');
+			strbuf_addch(sb, *s);
+			s++;
+		}
+		strbuf_addch(sb, '/'); /* double slashes to end prefix */
+	}
+	if (p->negative)
+		strbuf_addch(sb, '!');
+	if (p->has_root)
+		strbuf_addch(sb, '/');
+	else if (p->has_slashes && strchr(p->pattern, '/') == NULL) {
+		strbuf_addch(sb, '.');
+		strbuf_addch(sb, '/');
+	}
+	s = p->pattern;
+	while (*s) {
+		if (*s == ':')
+			strbuf_addch(sb, '\\');
+		strbuf_addch(sb, *s);
+		s++;
+	}
+}
+
 int match_narrow_spec(struct narrow_spec *spec, const char *path)
 {
 	int i, match;
diff --git a/unpack-trees.h b/unpack-trees.h
index 7e9febd..c5b19e7 100644
--- a/unpack-trees.h
+++ b/unpack-trees.h
@@ -75,6 +75,7 @@ extern int unpack_trees(unsigned n, struct tree_desc *t,
 
 struct narrow_spec *parse_narrow_spec(const char *spec, const char *prefix);
 void free_narrow_spec(struct narrow_spec *spec);
+void narrow_spec_to_strbuf(struct narrow_pattern *p, struct strbuf *sb);
 int match_narrow_spec(struct narrow_spec *spec, const char *path);
 int threeway_merge(struct cache_entry **stages, struct unpack_trees_options *o);
 int twoway_merge(struct cache_entry **src, struct unpack_trees_options *o);
-- 
1.6.0.3.890.g95457

[PATCH 8/8] wt-status: show sparse checkout info

[Nguyễn Thái Ngọc Duy]

This will make "git status" show core.defaultsparse and list of
orphaned files if present.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 wt-status.c |   44 ++++++++++++++++++++++++++++++++++++++++++++
 wt-status.h |    1 +
 2 files changed, 45 insertions(+), 0 deletions(-)

diff --git a/wt-status.c b/wt-status.c
index 3edae43..71cf809 100644
--- a/wt-status.c
+++ b/wt-status.c
@@ -20,6 +20,7 @@ static char wt_status_colors[][COLOR_MAXLEN] = {
 	"\033[31m", /* WT_STATUS_CHANGED: red */
 	"\033[31m", /* WT_STATUS_UNTRACKED: red */
 	"\033[31m", /* WT_STATUS_NOBRANCH: red */
+	"\033[31m", /* WT_STATUS_ORPHANED: red */
 };
 
 enum untracked_status_type show_untracked_files = SHOW_NORMAL_UNTRACKED_FILES;
@@ -83,6 +84,16 @@ static void wt_status_print_dirty_header(struct wt_status *s,
 	color_fprintf_ln(s->fp, c, "#");
 }
 
+static void wt_status_print_orphaned_header(struct wt_status *s)
+{
+	const char *c = color(WT_STATUS_HEADER);
+	color_fprintf_ln(s->fp, c, "# Orphaned files:");
+	color_fprintf_ln(s->fp, c, "#   (these are tracked, but marked no-checkout and should not be present)");
+	color_fprintf_ln(s->fp, c, "#   (use \"git update-index --checkout\" to remove no-checkout status)");
+	color_fprintf_ln(s->fp, c, "#   (otherwise remove them to avoid confusion because git will ignore them)");
+	color_fprintf_ln(s->fp, c, "#");
+}
+
 static void wt_status_print_untracked_header(struct wt_status *s)
 {
 	const char *c = color(WT_STATUS_HEADER);
@@ -211,6 +222,33 @@ static void wt_status_print_changed(struct wt_status *s)
 	run_diff_files(&rev, 0);
 }
 
+static void wt_status_print_orphaned(struct wt_status *s)
+{
+	int i, show_header = 0;
+	struct stat st;
+	struct strbuf buf;
+
+	strbuf_init(&buf, 0);
+	for (i = 0; i < the_index.cache_nr; i++) {
+		struct cache_entry *ce = the_index.cache[i];
+
+		if (ce_checkout(ce) || lstat(ce->name, &st))
+			continue;
+
+		if (!show_header) {
+			wt_status_print_orphaned_header(s);
+			show_header = 1;
+		}
+
+		color_fprintf(s->fp, color(WT_STATUS_HEADER), "#\t");
+		color_fprintf_ln(s->fp, color(WT_STATUS_ORPHANED), "%s",
+				 quote_path(ce->name, -1, &buf, s->prefix));
+	}
+
+	if (show_header)
+		color_fprintf_ln(s->fp, color(WT_STATUS_HEADER),"#");
+}
+
 static void wt_status_print_submodule_summary(struct wt_status *s)
 {
 	struct child_process sm_summary;
@@ -344,8 +382,14 @@ void wt_status_print(struct wt_status *s)
 		color_fprintf_ln(s->fp, color(WT_STATUS_HEADER), "#");
 	}
 
+	if (core_default_sparse) {
+		color_fprintf_ln(s->fp, color(WT_STATUS_HEADER), "# Sparse patterns: %s", core_default_sparse);
+		color_fprintf_ln(s->fp, color(WT_STATUS_HEADER), "#");
+	}
+
 	wt_status_print_updated(s);
 	wt_status_print_changed(s);
+	wt_status_print_orphaned(s);
 	if (wt_status_submodule_summary)
 		wt_status_print_submodule_summary(s);
 	if (show_untracked_files)
diff --git a/wt-status.h b/wt-status.h
index 78add09..52f1eb5 100644
--- a/wt-status.h
+++ b/wt-status.h
@@ -9,6 +9,7 @@ enum color_wt_status {
 	WT_STATUS_CHANGED,
 	WT_STATUS_UNTRACKED,
 	WT_STATUS_NOBRANCH,
+	WT_STATUS_ORPHANED,
 };
 
 enum untracked_status_type {
-- 
1.6.0.3.890.g95457

Re: [PATCH 1/8] generate-cmdlist.sh: avoid selecting synopsis at wrong place

[Johannes Schindelin]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 972 bytes --]

Hi,

On Sun, 30 Nov 2008, Nguyễn Thái Ngọc Duy wrote:

> In "common" man pages there is luckily no "NAME" anywhere except at 
> beginning of documents. If there is another "NAME", sed could mis-select 
> it and lead to common-cmds.h corruption. So better nail it at beginning 
> of line, which would reduce corruption chance.

I have no idea why you put this into the sparse checkout patch series.

As it is, the patch series is _already_ hard to review (as it is large not 
only in term of number of patches, but also individual patch size), 
_especially_ given the fact that there is no clear, precise and short 
description of why/how the sparse checkout is implemented.

For example, instead of using the BLURP area of the cover letter to put 
forth convincing arguments why this is needed, and why it is implemented 
in the best possible manner, you wasted two lines stating the obvious.

IMO that is almost like _asking_ people not to have a look at it.

Ciao,
Dscho

Re: [PATCH 1/8] generate-cmdlist.sh: avoid selecting synopsis at wrong place

[Nguyen Thai Ngoc Duy]

On 12/1/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> Hi,
>
>
>  On Sun, 30 Nov 2008, Nguy­n Thái Ng÷c Duy wrote:
>
>  > In "common" man pages there is luckily no "NAME" anywhere except at
>  > beginning of documents. If there is another "NAME", sed could mis-select
>  > it and lead to common-cmds.h corruption. So better nail it at beginning
>  > of line, which would reduce corruption chance.
>
>
> I have no idea why you put this into the sparse checkout patch series.

It's more an independent bug fix. I posted this once but it was not
caught up. I intended to just put some comments in BLURB area to tell
people to get this patch from mail archive. But that way seems
inconvenient. Without this patch, you won't be able to compile the
series because of common-cmds.h corruption.
-- 
Duy

Re: [PATCH 1/8] generate-cmdlist.sh: avoid selecting synopsis at wrong place

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> On Sun, 30 Nov 2008, Nguyễn Thái Ngọc Duy wrote:
>
>> In "common" man pages there is luckily no "NAME" anywhere except at 
>> beginning of documents. If there is another "NAME", sed could mis-select 
>> it and lead to common-cmds.h corruption. So better nail it at beginning 
>> of line, which would reduce corruption chance.
>
> I have no idea why you put this into the sparse checkout patch series.

That is because a documentation that triggers the misfortune this patch
fixes is introduced in the series.  I actually scratched head myself,
though, and come to think of it, I should have complained that this patch
should state that because it does not look like it has anything do with
the main topic of the series.

> As it is, the patch series is _already_ hard to review (as it is large not 
> only in term of number of patches, but also individual patch size), 
> _especially_ given the fact that there is no clear, precise and short 
> description of why/how the sparse checkout is implemented.

Hmm, can you really tell the lack of such description without reading the
series, I have to wonder...

Re: [PATCH 1/8] generate-cmdlist.sh: avoid selecting synopsis at wrong place

[Johannes Schindelin]

Hi,

On Mon, 1 Dec 2008, Junio C Hamano wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
> > As it is, the patch series is _already_ hard to review (as it is large 
> > not only in term of number of patches, but also individual patch 
> > size), _especially_ given the fact that there is no clear, precise and 
> > short description of why/how the sparse checkout is implemented.
> 
> Hmm, can you really tell the lack of such description without reading 
> the series, I have to wonder...

Okay, I thought it was obvious, but here is a template for the BLURP of 
the cover letter that would at least get me started:

-- snip --
A "sparse checkout" is an index/working directory pair where not all 
files/directories of the HEAD commit are actually checked out in the 
working directory.  Instead, they are marked as "not being checked out" in 
the index.

The real meat of this series is patch *** M/N *** which teaches Git to 
understand the *** XYZ flag *** for index entries.

The following operations are affected by sparse checkout: *** X, Y, Z ***

These operations respect sparse checkouts by *** THIS, THIS AND THIS ***.

The first patch really should be independent, but patch *** M/N *** would 
fail without it.
-- snap --

And of course, the whole BLURP should not consist of 10-20 line 
paragraphs, but try to fit everything into 3-4 line paragraphs (I seem to 
remember that there was a mail on this list saying than more than 4 
lines/paragraph are too much for the average attention span...).

Ciao,
Dscho