[PATCH v2 00/14] Sparse checkout

[PATCH v2 00/14] Sparse checkout

[Nguyễn Thái Ngọc Duy]

Main changes from the last round are:
 - The name is now "sparse checkout"
 - "git clone --path" => "git clone --narrow-path"
 - "git checkout --path" => "git checkout --reset-path"
 - New narrow spec (or "sparse patterns" from now) resembles
   .gitignore patterns
 - "git ls-files" now supports more fine-grained listing. It can now
   list checkout files, no-checkout files or orphaned (previously
   "overlay") files. --overlay is gone
 - "git status" shows orphaned entries and remedies
 - Documentation has been restructured to accompany code changes.
   Thanks to Jakub, Baz for lots of input.

For code changes, significant changes are:
  [03/14] ls-files: add options to support sparse checkout
  [10/14] ls-files: support "sparse patterns", used to form sparse checkout areas

I hope I have addressed all the issues. If I miss anything, please speak up.

Nguyễn Thái Ngọc Duy (14):
  Extend index to save more flags
  Introduce CE_NO_CHECKOUT bit
  ls-files: add options to support sparse checkout
  update-index: refactor mark_valid() in preparation for new options
  update-index: add --checkout/--no-checkout to update CE_NO_CHECKOUT bit
  ls-files: Add tests for --sparse and friends
  Prevent diff machinery from examining worktree outside sparse checkout
  checkout_entry(): CE_NO_CHECKOUT on checked out entries.
  grep: skip files outside sparse checkout area
  ls-files: support "sparse patterns", used to form sparse checkout areas
  unpack_trees(): add support for sparse checkout
  clone: support sparse checkout with --narrow-path option
  checkout: add new options to support sparse checkout
  wt-status: Show orphaned entries in "git status" output

 .gitignore                            |    1 +
 Documentation/git-checkout.txt        |  131 ++++++++++++++++++++-
 Documentation/git-clone.txt           |   10 ++-
 Documentation/git-grep.txt            |    4 +-
 Documentation/git-ls-files.txt        |   30 +++++-
 Documentation/git-update-index.txt    |   13 ++
 Makefile                              |    2 +-
 builtin-checkout.c                    |   37 ++++++
 builtin-clone.c                       |   13 ++
 builtin-grep.c                        |    7 +-
 builtin-ls-files.c                    |   60 +++++++++-
 builtin-update-index.c                |   40 ++++---
 cache.h                               |   69 ++++++++++-
 diff-lib.c                            |    5 +-
 diff.c                                |    4 +-
 entry.c                               |    1 +
 read-cache.c                          |   57 +++++++--
 t/t2011-checkout-sparse.sh            |  108 +++++++++++++++++
 t/t2104-update-index-no-checkout.sh   |   36 ++++++
 t/t3003-ls-files-narrow-match.sh      |   39 ++++++
 t/t3003/1                             |    3 +
 t/t3003/12                            |    6 +
 t/t3003/clone-escape                  |    4 +
 t/t3003/cur-12                        |    2 +
 t/t3003/root-sub-1                    |    1 +
 t/t3003/slash-1                       |    1 +
 t/t3003/sub-1                         |    2 +
 t/t3003/sub-only                      |    3 +
 t/t3003/subsub-slash                  |    3 +
 t/t3004-ls-files-sparse.sh            |   40 ++++++
 t/t3004/cached.expected               |    5 +
 t/t3004/deleted.expected              |    1 +
 t/t3004/everything.expected           |   10 ++
 t/t3004/modified.expected             |    2 +
 t/t3004/no-checkout.expected          |    2 +
 t/t3004/orphaned-no-checkout.expected |    3 +
 t/t3004/orphaned.expected             |    1 +
 t/t3004/others.expected               |    2 +
 t/t3004/sparse-cached.expected        |    3 +
 t/t3004/sparse-everything.expected    |   11 ++
 t/t5703-clone-narrow.sh               |   39 ++++++
 test-index-version.c                  |   14 ++
 unpack-trees.c                        |  210 ++++++++++++++++++++++++++++++++-
 unpack-trees.h                        |   22 ++++
 wt-status.c                           |   39 ++++++
 wt-status.h                           |    1 +
 46 files changed, 1047 insertions(+), 50 deletions(-)
 create mode 100755 t/t2011-checkout-sparse.sh
 create mode 100755 t/t2104-update-index-no-checkout.sh
 create mode 100755 t/t3003-ls-files-narrow-match.sh
 create mode 100644 t/t3003/1
 create mode 100644 t/t3003/12
 create mode 100644 t/t3003/clone-escape
 create mode 100644 t/t3003/cur-12
 create mode 100644 t/t3003/root-sub-1
 create mode 100644 t/t3003/slash-1
 create mode 100644 t/t3003/sub
 create mode 100644 t/t3003/sub-1
 create mode 100644 t/t3003/sub-only
 create mode 100644 t/t3003/subsub-slash
 create mode 100755 t/t3004-ls-files-sparse.sh
 create mode 100644 t/t3004/cached.expected
 create mode 100644 t/t3004/deleted.expected
 create mode 100644 t/t3004/everything.expected
 create mode 100644 t/t3004/modified.expected
 create mode 100644 t/t3004/no-checkout.expected
 create mode 100644 t/t3004/orphaned-no-checkout.expected
 create mode 100644 t/t3004/orphaned.expected
 create mode 100644 t/t3004/others.expected
 create mode 100644 t/t3004/sparse-cached.expected
 create mode 100644 t/t3004/sparse-everything.expected
 create mode 100755 t/t5703-clone-narrow.sh
 create mode 100644 test-index-version.c

[PATCH 01/14] Extend index to save more flags

[Nguyễn Thái Ngọc Duy]

The on-disk format of index only saves 16 bit flags, nearly all have
been used. The last bit (CE_EXTENDED) is used to for future extension.

This patch extends index entry format to save more flags in future.
The new entry format will be used when CE_EXTENDED bit is 1.

Because older implementation may not understand CE_EXTENDED bit and
misread the new format, if there is any extended entry in index, index
header version will turn 3, which makes it incompatible for older git.
If there is none, header version will return to 2 again.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 cache.h      |   58 ++++++++++++++++++++++++++++++++++++++++++++++++++++++----
 read-cache.c |   51 +++++++++++++++++++++++++++++++++++++++++----------
 2 files changed, 95 insertions(+), 14 deletions(-)

diff --git a/cache.h b/cache.h
index f4b8ddf..77b6eb3 100644
--- a/cache.h
+++ b/cache.h
@@ -109,6 +109,26 @@ struct ondisk_cache_entry {
 	char name[FLEX_ARRAY]; /* more */
 };
 
+/*
+ * This struct is used when CE_EXTENDED bit is 1
+ * The struct must match ondisk_cache_entry exactly from
+ * ctime till flags
+ */
+struct ondisk_cache_entry_extended {
+	struct cache_time ctime;
+	struct cache_time mtime;
+	unsigned int dev;
+	unsigned int ino;
+	unsigned int mode;
+	unsigned int uid;
+	unsigned int gid;
+	unsigned int size;
+	unsigned char sha1[20];
+	unsigned short flags;
+	unsigned short flags2;
+	char name[FLEX_ARRAY]; /* more */
+};
+
 struct cache_entry {
 	unsigned int ce_ctime;
 	unsigned int ce_mtime;
@@ -130,7 +150,15 @@ struct cache_entry {
 #define CE_VALID     (0x8000)
 #define CE_STAGESHIFT 12
 
-/* In-memory only */
+/*
+ * Range 0xFFFF0000 in ce_flags is divided into
+ * two parts: in-memory flags and on-disk ones.
+ * Flags in CE_EXTENDED_FLAGS will get saved on-disk
+ * if you want to save a new flag, add it in
+ * CE_EXTENDED_FLAGS
+ *
+ * In-memory only flags
+ */
 #define CE_UPDATE    (0x10000)
 #define CE_REMOVE    (0x20000)
 #define CE_UPTODATE  (0x40000)
@@ -140,6 +168,24 @@ struct cache_entry {
 #define CE_UNHASHED  (0x200000)
 
 /*
+ * Extended on-disk flags
+ */
+/* CE_EXTENDED2 is for future extension */
+#define CE_EXTENDED2 0x80000000
+
+#define CE_EXTENDED_FLAGS (0)
+
+/*
+ * Safeguard to avoid saving wrong flags:
+ *  - CE_EXTENDED2 won't get saved until its semantic is known
+ *  - Bits in 0x0000FFFF have been saved in ce_flags already
+ *  - Bits in 0x003F0000 are currently in-memory flags
+ */
+#if CE_EXTENDED_FLAGS & 0x80CFFFFF
+#error "CE_EXTENDED_FLAGS out of range"
+#endif
+
+/*
  * Copy the sha1 and stat state of a cache entry from one to
  * another. But we never change the name, or the hash state!
  */
@@ -171,7 +217,9 @@ static inline size_t ce_namelen(const struct cache_entry *ce)
 }
 
 #define ce_size(ce) cache_entry_size(ce_namelen(ce))
-#define ondisk_ce_size(ce) ondisk_cache_entry_size(ce_namelen(ce))
+#define ondisk_ce_size(ce) (((ce)->ce_flags & CE_EXTENDED) ? \
+			    ondisk_cache_entry_extended_size(ce_namelen(ce)) : \
+			    ondisk_cache_entry_size(ce_namelen(ce)))
 #define ce_stage(ce) ((CE_STAGEMASK & (ce)->ce_flags) >> CE_STAGESHIFT)
 #define ce_uptodate(ce) ((ce)->ce_flags & CE_UPTODATE)
 #define ce_mark_uptodate(ce) ((ce)->ce_flags |= CE_UPTODATE)
@@ -214,8 +262,10 @@ static inline int ce_to_dtype(const struct cache_entry *ce)
 	(S_ISREG(mode) ? (S_IFREG | ce_permissions(mode)) : \
 	S_ISLNK(mode) ? S_IFLNK : S_ISDIR(mode) ? S_IFDIR : S_IFGITLINK)
 
-#define cache_entry_size(len) ((offsetof(struct cache_entry,name) + (len) + 8) & ~7)
-#define ondisk_cache_entry_size(len) ((offsetof(struct ondisk_cache_entry,name) + (len) + 8) & ~7)
+#define flexible_size(STRUCT,len) ((offsetof(struct STRUCT,name) + (len) + 8) & ~7)
+#define cache_entry_size(len) flexible_size(cache_entry,len)
+#define ondisk_cache_entry_size(len) flexible_size(ondisk_cache_entry,len)
+#define ondisk_cache_entry_extended_size(len) flexible_size(ondisk_cache_entry_extended,len)
 
 struct index_state {
 	struct cache_entry **cache;
diff --git a/read-cache.c b/read-cache.c
index c5a8659..667c36b 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -1096,7 +1096,7 @@ static int verify_hdr(struct cache_header *hdr, unsigned long size)
 
 	if (hdr->hdr_signature != htonl(CACHE_SIGNATURE))
 		return error("bad signature");
-	if (hdr->hdr_version != htonl(2))
+	if (hdr->hdr_version != htonl(2) && hdr->hdr_version != htonl(3))
 		return error("bad index version");
 	SHA1_Init(&c);
 	SHA1_Update(&c, hdr, size - 20);
@@ -1131,6 +1131,7 @@ int read_index(struct index_state *istate)
 static void convert_from_disk(struct ondisk_cache_entry *ondisk, struct cache_entry *ce)
 {
 	size_t len;
+	const char *name;
 
 	ce->ce_ctime = ntohl(ondisk->ctime.sec);
 	ce->ce_mtime = ntohl(ondisk->mtime.sec);
@@ -1143,19 +1144,31 @@ static void convert_from_disk(struct ondisk_cache_entry *ondisk, struct cache_en
 	/* On-disk flags are just 16 bits */
 	ce->ce_flags = ntohs(ondisk->flags);
 
-	/* For future extension: we do not understand this entry yet */
-	if (ce->ce_flags & CE_EXTENDED)
-		die("Unknown index entry format");
 	hashcpy(ce->sha1, ondisk->sha1);
 
 	len = ce->ce_flags & CE_NAMEMASK;
+
+	if (ce->ce_flags & CE_EXTENDED) {
+		struct ondisk_cache_entry_extended *ondisk2;
+		int extended_flags;
+		ondisk2 = (struct ondisk_cache_entry_extended *)ondisk;
+		extended_flags = ntohs(ondisk2->flags2) << 16;
+		/* We do not yet understand any bit out of CE_EXTENDED_FLAGS */
+		if (extended_flags & ~CE_EXTENDED_FLAGS)
+			die("Unknown index entry format %08x", extended_flags);
+		ce->ce_flags |= extended_flags;
+		name = ondisk2->name;
+	}
+	else
+		name = ondisk->name;
+
 	if (len == CE_NAMEMASK)
-		len = strlen(ondisk->name);
+		len = strlen(name);
 	/*
 	 * NEEDSWORK: If the original index is crafted, this copy could
 	 * go unchecked.
 	 */
-	memcpy(ce->name, ondisk->name, len + 1);
+	memcpy(ce->name, name, len + 1);
 }
 
 static inline size_t estimate_cache_size(size_t ondisk_size, unsigned int entries)
@@ -1415,6 +1428,7 @@ static int ce_write_entry(SHA_CTX *c, int fd, struct cache_entry *ce)
 {
 	int size = ondisk_ce_size(ce);
 	struct ondisk_cache_entry *ondisk = xcalloc(1, size);
+	char *name;
 
 	ondisk->ctime.sec = htonl(ce->ce_ctime);
 	ondisk->ctime.nsec = 0;
@@ -1428,7 +1442,15 @@ static int ce_write_entry(SHA_CTX *c, int fd, struct cache_entry *ce)
 	ondisk->size = htonl(ce->ce_size);
 	hashcpy(ondisk->sha1, ce->sha1);
 	ondisk->flags = htons(ce->ce_flags);
-	memcpy(ondisk->name, ce->name, ce_namelen(ce));
+	if (ce->ce_flags & CE_EXTENDED) {
+		struct ondisk_cache_entry_extended *ondisk2;
+		ondisk2 = (struct ondisk_cache_entry_extended *)ondisk;
+		ondisk2->flags2 = htons((ce->ce_flags & CE_EXTENDED_FLAGS) >> 16);
+		name = ondisk2->name;
+	}
+	else
+		name = ondisk->name;
+	memcpy(name, ce->name, ce_namelen(ce));
 
 	return ce_write(c, fd, ondisk, size);
 }
@@ -1437,16 +1459,25 @@ int write_index(const struct index_state *istate, int newfd)
 {
 	SHA_CTX c;
 	struct cache_header hdr;
-	int i, err, removed;
+	int i, err, removed, extended;
 	struct cache_entry **cache = istate->cache;
 	int entries = istate->cache_nr;
 
-	for (i = removed = 0; i < entries; i++)
+	for (i = removed = extended = 0; i < entries; i++) {
 		if (cache[i]->ce_flags & CE_REMOVE)
 			removed++;
 
+		/* reduce extended entries if possible */
+		cache[i]->ce_flags &= ~CE_EXTENDED;
+		if (cache[i]->ce_flags & CE_EXTENDED_FLAGS) {
+			extended++;
+			cache[i]->ce_flags |= CE_EXTENDED;
+		}
+	}
+
 	hdr.hdr_signature = htonl(CACHE_SIGNATURE);
-	hdr.hdr_version = htonl(2);
+	/* for extended format, increase version so older git won't try to read it */
+	hdr.hdr_version = htonl(extended ? 3 : 2);
 	hdr.hdr_entries = htonl(entries - removed);
 
 	SHA1_Init(&c);
-- 
1.6.0.96.g2fad1.dirty

[PATCH 02/14] Introduce CE_NO_CHECKOUT bit

[Nguyễn Thái Ngọc Duy]

This bit is the basis of sparse checkout. If this bit is on, the entry
is outside sparse checkout and therefore should be ignored (similar
to CE_VALID)

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-checkout.txt |   33 +++++++++++++++++++++++++++++++++
 cache.h                        |   10 +++++++++-
 read-cache.c                   |    6 +++---
 3 files changed, 45 insertions(+), 4 deletions(-)

diff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt
index 82e154d..4bd9eba 100644
--- a/Documentation/git-checkout.txt
+++ b/Documentation/git-checkout.txt
@@ -171,6 +171,39 @@ the reflog for HEAD where you were, e.g.
 $ git log -g -2 HEAD
 ------------
 
+Sparse checkout
+---------------
+
+Normally when you checkout a branch, your working directory
+will be fully populated. In some situations, you just need to
+work on certain files, no full checkout is needed. Sparse
+checkout is a mode that limits the checkout area according to your
+needs. With sparse checkout, you can work on a single file, a
+collection of files, a subdirectory or a collection of separated
+subdirectories.
+
+Because sparse checkout uses a new index format, it will be
+incompatible with git prior to 1.6.0 regarding worktree operations.
+Operations that only need access to the repository itself, such as
+clone, push, or pull/fetch from another (normal) repository... should
+not be affected by sparse checkout.
+
+In sparse checkout mode, checkout status of every files in your
+working directory will be recorded in index. If a file is marked
+"no-checkout", it means that file is not needed to be present in
+working directory by user or any git command. When a new file is added
+to index, it will be marked "checkout" unless sparse patterns are
+applied.  Unmerged files are always "checkout". When you checkout new
+files using "git checkout <file>" they will be automatically marked
+"checkout". Other commands such as "git apply" can also checkout new
+files if they are needed.
+
+"No-checkout" status is very similar to "assume-unchanged bit"
+(see linkgit:git-update-index[1]). The main difference between them
+is "assume unchanged" bit just ignores corresponding files in working
+directory while sparse checkout goes a bit farther, remove those files
+when it is safe to do so.
+
 
 EXAMPLES
 --------
diff --git a/cache.h b/cache.h
index 77b6eb3..6e875d5 100644
--- a/cache.h
+++ b/cache.h
@@ -170,10 +170,11 @@ struct cache_entry {
 /*
  * Extended on-disk flags
  */
+#define CE_NO_CHECKOUT 0x40000000
 /* CE_EXTENDED2 is for future extension */
 #define CE_EXTENDED2 0x80000000
 
-#define CE_EXTENDED_FLAGS (0)
+#define CE_EXTENDED_FLAGS (CE_NO_CHECKOUT)
 
 /*
  * Safeguard to avoid saving wrong flags:
@@ -185,6 +186,9 @@ struct cache_entry {
 #error "CE_EXTENDED_FLAGS out of range"
 #endif
 
+/* "Assume unchanged" mask */
+#define CE_VALID_MASK (CE_VALID | CE_NO_CHECKOUT)
+
 /*
  * Copy the sha1 and stat state of a cache entry from one to
  * another. But we never change the name, or the hash state!
@@ -222,6 +226,10 @@ static inline size_t ce_namelen(const struct cache_entry *ce)
 			    ondisk_cache_entry_size(ce_namelen(ce)))
 #define ce_stage(ce) ((CE_STAGEMASK & (ce)->ce_flags) >> CE_STAGESHIFT)
 #define ce_uptodate(ce) ((ce)->ce_flags & CE_UPTODATE)
+#define ce_no_checkout(ce) ((ce)->ce_flags & CE_NO_CHECKOUT)
+#define ce_checkout(ce) (!ce_no_checkout(ce))
+#define ce_mark_no_checkout(ce) ((ce)->ce_flags |= CE_NO_CHECKOUT)
+#define ce_mark_checkout(ce) ((ce)->ce_flags &= ~CE_NO_CHECKOUT)
 #define ce_mark_uptodate(ce) ((ce)->ce_flags |= CE_UPTODATE)
 
 #define ce_permissions(mode) (((mode) & 0100) ? 0755 : 0644)
diff --git a/read-cache.c b/read-cache.c
index 667c36b..e965a4c 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -254,7 +254,7 @@ int ie_match_stat(const struct index_state *istate,
 	 * If it's marked as always valid in the index, it's
 	 * valid whatever the checked-out copy says.
 	 */
-	if (!ignore_valid && (ce->ce_flags & CE_VALID))
+	if (!ignore_valid && (ce->ce_flags & CE_VALID_MASK))
 		return 0;
 
 	changed = ce_match_stat_basic(ce, st);
@@ -962,10 +962,10 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,
 		return ce;
 
 	/*
-	 * CE_VALID means the user promised us that the change to
+	 * CE_VALID_MASK means the user promised us that the change to
 	 * the work tree does not matter and told us not to worry.
 	 */
-	if (!ignore_valid && (ce->ce_flags & CE_VALID)) {
+	if (!ignore_valid && (ce->ce_flags & CE_VALID_MASK)) {
 		ce_mark_uptodate(ce);
 		return ce;
 	}
-- 
1.6.0.96.g2fad1.dirty

[PATCH 03/14] ls-files: add options to support sparse checkout

[Nguyễn Thái Ngọc Duy]

The first option to be introduced is --sparse, which puts ls-files
in "sparse mode". In this mode, cached entries are divided into
 - checkout entries: shown by --cached (new behavior with --sparse)
 - no-checkout entries: show by --no-checkout (new option)
 - orphaned entries: shown by --orphaned (new option)

Orphaned entries are themselves no-checkout ones but for some reasons
still be present in working directory.

While at it, fix "--deleted" running out of checkout area.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-ls-files.txt |   24 +++++++++++++++++++++-
 builtin-ls-files.c             |   41 ++++++++++++++++++++++++++++++++++++---
 2 files changed, 59 insertions(+), 6 deletions(-)

diff --git a/Documentation/git-ls-files.txt b/Documentation/git-ls-files.txt
index 9f85d60..1de68e2 100644
--- a/Documentation/git-ls-files.txt
+++ b/Documentation/git-ls-files.txt
@@ -10,8 +10,9 @@ SYNOPSIS
 --------
 [verse]
 'git ls-files' [-z] [-t] [-v]
-		(--[cached|deleted|others|ignored|stage|unmerged|killed|modified])\*
+		(--[cached|deleted|others|ignored|stage|unmerged|killed|modified|orphaned|no-checkout])\*
 		(-[c|d|o|i|s|u|k|m])\*
+		[--sparse]
 		[-x <pattern>|--exclude=<pattern>]
 		[-X <file>|--exclude-from=<file>]
 		[--exclude-per-directory=<file>]
@@ -32,7 +33,9 @@ OPTIONS
 -------
 -c::
 --cached::
-	Show cached files in the output (default)
+	Show cached files in the output (default). When used with --sparse,
+	show only cached files that are marked "checkout", no-checkout
+	entries will be excluded.
 
 -d::
 --deleted::
@@ -72,6 +75,21 @@ OPTIONS
 	to file/directory conflicts for checkout-index to
 	succeed.
 
+--no-checkout::
+	Show no-checkout entries. This option implies --sparse.
+
+--orphaned::
+	Show orphaned entries. Orphaned entries are no-checkout
+	entries that are present in working directory. This option
+	implies --sparse.
+
+--sparse::
+	When --sparse is passed, cached files will be divided into two
+	parts: checkout entries and no-checkout entries.
+	--cached will only show checkout entries.
+	No-checkout entries can be shown using --orphaned or
+	--no-checkout (or both).
+
 -z::
 	\0 line termination on output.
 
@@ -107,6 +125,8 @@ OPTIONS
 	Identify the file status with the following tags (followed by
 	a space) at the start of each line:
 	H::	cached
+	-::	no-checkout entries
+	O::	orphaned entries
 	M::	unmerged
 	R::	removed/deleted
 	C::	modified/changed
diff --git a/builtin-ls-files.c b/builtin-ls-files.c
index 068f424..873de15 100644
--- a/builtin-ls-files.c
+++ b/builtin-ls-files.c
@@ -20,6 +20,9 @@ static int show_unmerged;
 static int show_modified;
 static int show_killed;
 static int show_valid_bit;
+static int show_orphaned;
+static int show_no_checkout;
+static int sparse_checkout;
 static int line_terminator = '\n';
 
 static int prefix_len;
@@ -35,6 +38,8 @@ static const char *tag_removed = "";
 static const char *tag_other = "";
 static const char *tag_killed = "";
 static const char *tag_modified = "";
+static const char *tag_orphaned = "";
+static const char *tag_no_checkout = "";
 
 
 /*
@@ -235,7 +240,7 @@ static void show_files(struct dir_struct *dir, const char *prefix)
 		if (show_killed)
 			show_killed_files(dir);
 	}
-	if (show_cached | show_stage) {
+	if (show_cached | show_stage | show_orphaned | show_no_checkout) {
 		for (i = 0; i < active_nr; i++) {
 			struct cache_entry *ce = active_cache[i];
 			int dtype = ce_to_dtype(ce);
@@ -245,6 +250,16 @@ static void show_files(struct dir_struct *dir, const char *prefix)
 				continue;
 			if (ce->ce_flags & CE_UPDATE)
 				continue;
+			if (sparse_checkout && ce_no_checkout(ce)) {
+				struct stat st;
+				if (show_no_checkout)
+					show_ce_entry(tag_no_checkout, ce);
+				if (show_orphaned && !lstat(ce->name, &st))
+					show_ce_entry(tag_orphaned, ce);
+				continue;
+			}
+			if (!(show_cached | show_stage))
+				continue;
 			show_ce_entry(ce_stage(ce) ? tag_unmerged : tag_cached, ce);
 		}
 	}
@@ -257,7 +272,7 @@ static void show_files(struct dir_struct *dir, const char *prefix)
 			if (excluded(dir, ce->name, &dtype) != dir->show_ignored)
 				continue;
 			err = lstat(ce->name, &st);
-			if (show_deleted && err)
+			if (show_deleted && err && ce_checkout(ce))
 				show_ce_entry(tag_removed, ce);
 			if (show_modified && ce_modified(ce, &st, 0))
 				show_ce_entry(tag_modified, ce);
@@ -423,7 +438,8 @@ int report_path_error(const char *ps_matched, const char **pathspec, int prefix_
 }
 
 static const char ls_files_usage[] =
-	"git ls-files [-z] [-t] [-v] (--[cached|deleted|others|stage|unmerged|killed|modified])* "
+	"git ls-files [-z] [-t] [-v] (--[cached|deleted|others|stage|unmerged|killed|modified|orphaned|no-checkout])* "
+	"[ --sparse ] "
 	"[ --ignored ] [--exclude=<pattern>] [--exclude-from=<file>] "
 	"[ --exclude-per-directory=<filename> ] [--exclude-standard] "
 	"[--full-name] [--abbrev] [--] [<file>]*";
@@ -457,6 +473,8 @@ int cmd_ls_files(int argc, const char **argv, const char *prefix)
 			tag_modified = "C ";
 			tag_other = "? ";
 			tag_killed = "K ";
+			tag_orphaned = "O ";
+			tag_no_checkout = "- ";
 			if (arg[1] == 'v')
 				show_valid_bit = 1;
 			continue;
@@ -465,6 +483,21 @@ int cmd_ls_files(int argc, const char **argv, const char *prefix)
 			show_cached = 1;
 			continue;
 		}
+		if (!strcmp(arg, "--sparse")) {
+			sparse_checkout = 1;
+			continue;
+		}
+		if (!strcmp(arg, "--orphaned")) {
+			show_orphaned = 1;
+			sparse_checkout = 1;
+			require_work_tree = 1;
+			continue;
+		}
+		if (!strcmp(arg, "--no-checkout")) {
+			show_no_checkout = 1;
+			sparse_checkout = 1;
+			continue;
+		}
 		if (!strcmp(arg, "-d") || !strcmp(arg, "--deleted")) {
 			show_deleted = 1;
 			continue;
@@ -593,7 +626,7 @@ int cmd_ls_files(int argc, const char **argv, const char *prefix)
 
 	/* With no flags, we default to showing the cached files */
 	if (!(show_stage | show_deleted | show_others | show_unmerged |
-	      show_killed | show_modified))
+	      show_killed | show_modified | show_orphaned | show_no_checkout))
 		show_cached = 1;
 
 	read_cache();
-- 
1.6.0.96.g2fad1.dirty

[PATCH 04/14] update-index: refactor mark_valid() in preparation for new options

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin-update-index.c |   24 ++++++++++--------------
 1 files changed, 10 insertions(+), 14 deletions(-)

diff --git a/builtin-update-index.c b/builtin-update-index.c
index 417f972..ae94739 100644
--- a/builtin-update-index.c
+++ b/builtin-update-index.c
@@ -24,8 +24,8 @@ static int info_only;
 static int force_remove;
 static int verbose;
 static int mark_valid_only;
-#define MARK_VALID 1
-#define UNMARK_VALID 2
+#define MARK_FLAG 1
+#define UNMARK_FLAG 2
 
 static void report(const char *fmt, ...)
 {
@@ -40,19 +40,15 @@ static void report(const char *fmt, ...)
 	va_end(vp);
 }
 
-static int mark_valid(const char *path)
+static int mark_ce_flags(const char *path, int flag, int mark)
 {
 	int namelen = strlen(path);
 	int pos = cache_name_pos(path, namelen);
 	if (0 <= pos) {
-		switch (mark_valid_only) {
-		case MARK_VALID:
-			active_cache[pos]->ce_flags |= CE_VALID;
-			break;
-		case UNMARK_VALID:
-			active_cache[pos]->ce_flags &= ~CE_VALID;
-			break;
-		}
+		if (mark)
+			active_cache[pos]->ce_flags |= flag;
+		else
+			active_cache[pos]->ce_flags &= ~flag;
 		cache_tree_invalidate_path(active_cache_tree, path);
 		active_cache_changed = 1;
 		return 0;
@@ -276,7 +272,7 @@ static void update_one(const char *path, const char *prefix, int prefix_length)
 		goto free_return;
 	}
 	if (mark_valid_only) {
-		if (mark_valid(p))
+		if (mark_ce_flags(p, CE_VALID, mark_valid_only == MARK_FLAG))
 			die("Unable to mark file %s", path);
 		goto free_return;
 	}
@@ -649,11 +645,11 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)
 				continue;
 			}
 			if (!strcmp(path, "--assume-unchanged")) {
-				mark_valid_only = MARK_VALID;
+				mark_valid_only = MARK_FLAG;
 				continue;
 			}
 			if (!strcmp(path, "--no-assume-unchanged")) {
-				mark_valid_only = UNMARK_VALID;
+				mark_valid_only = UNMARK_FLAG;
 				continue;
 			}
 			if (!strcmp(path, "--info-only")) {
-- 
1.6.0.96.g2fad1.dirty

[PATCH 05/14] update-index: add --checkout/--no-checkout to update CE_NO_CHECKOUT bit

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 .gitignore                          |    1 +
 Documentation/git-checkout.txt      |    3 +-
 Documentation/git-update-index.txt  |   13 ++++++++++++
 Makefile                            |    2 +-
 builtin-update-index.c              |   16 ++++++++++++++-
 t/t2104-update-index-no-checkout.sh |   36 +++++++++++++++++++++++++++++++++++
 test-index-version.c                |   14 +++++++++++++
 7 files changed, 82 insertions(+), 3 deletions(-)
 create mode 100755 t/t2104-update-index-no-checkout.sh
 create mode 100644 test-index-version.c

diff --git a/.gitignore b/.gitignore
index bbaf9de..0c35577 100644
--- a/.gitignore
+++ b/.gitignore
@@ -147,6 +147,7 @@ test-date
 test-delta
 test-dump-cache-tree
 test-genrandom
+test-index-version
 test-match-trees
 test-parse-options
 test-path-utils
diff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt
index 4bd9eba..2b344e1 100644
--- a/Documentation/git-checkout.txt
+++ b/Documentation/git-checkout.txt
@@ -196,7 +196,8 @@ to index, it will be marked "checkout" unless sparse patterns are
 applied.  Unmerged files are always "checkout". When you checkout new
 files using "git checkout <file>" they will be automatically marked
 "checkout". Other commands such as "git apply" can also checkout new
-files if they are needed.
+files if they are needed. linkgit:git-update-index[1] can be used to
+update "checkout/no-checkout" status in index.
 
 "No-checkout" status is very similar to "assume-unchanged bit"
 (see linkgit:git-update-index[1]). The main difference between them
diff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt
index 1d9d81a..ec03e05 100644
--- a/Documentation/git-update-index.txt
+++ b/Documentation/git-update-index.txt
@@ -15,6 +15,7 @@ SYNOPSIS
 	     [--cacheinfo <mode> <object> <file>]\*
 	     [--chmod=(+|-)x]
 	     [--assume-unchanged | --no-assume-unchanged]
+	     [--checkout | --no-checkout]
 	     [--ignore-submodules]
 	     [--really-refresh] [--unresolve] [--again | -g]
 	     [--info-only] [--index-info]
@@ -99,6 +100,18 @@ in the index e.g. when merging in a commit;
 thus, in case the assumed-untracked file is changed upstream,
 you will need to handle the situation manually.
 
+--checkout::
+--no-checkout::
+	When one of these flags is specified, the object name recorded
+	for the paths are not updated. Instead, these options
+	set and unset the "no-checkout" bit for the paths. This
+	bit is used for marking files for sparse checkout. If
+	a path is marked "no-checkout", then it should not be
+	checked out unless requested by user or needed for a git
+	command to function.
+	See linkgit:git-checkout[1] for more information about
+	sparse checkout.
+
 -g::
 --again::
 	Runs 'git-update-index' itself on the paths whose index
diff --git a/Makefile b/Makefile
index e0c03c3..edb33cb 100644
--- a/Makefile
+++ b/Makefile
@@ -1327,7 +1327,7 @@ endif
 
 ### Testing rules
 
-TEST_PROGRAMS = test-chmtime$X test-genrandom$X test-date$X test-delta$X test-sha1$X test-match-trees$X test-parse-options$X test-path-utils$X
+TEST_PROGRAMS = test-chmtime$X test-genrandom$X test-date$X test-delta$X test-sha1$X test-match-trees$X test-parse-options$X test-path-utils$X test-index-version$X
 
 all:: $(TEST_PROGRAMS)
 
diff --git a/builtin-update-index.c b/builtin-update-index.c
index ae94739..7514aff 100644
--- a/builtin-update-index.c
+++ b/builtin-update-index.c
@@ -24,6 +24,7 @@ static int info_only;
 static int force_remove;
 static int verbose;
 static int mark_valid_only;
+static int mark_no_checkout_only;
 #define MARK_FLAG 1
 #define UNMARK_FLAG 2
 
@@ -276,6 +277,11 @@ static void update_one(const char *path, const char *prefix, int prefix_length)
 			die("Unable to mark file %s", path);
 		goto free_return;
 	}
+	if (mark_no_checkout_only) {
+		if (mark_ce_flags(p, CE_NO_CHECKOUT, mark_no_checkout_only == MARK_FLAG))
+			die("Unable to mark file %s", path);
+		goto free_return;
+	}
 
 	if (force_remove) {
 		if (remove_file_from_cache(p))
@@ -386,7 +392,7 @@ static void read_index_info(int line_termination)
 }
 
 static const char update_index_usage[] =
-"git update-index [-q] [--add] [--replace] [--remove] [--unmerged] [--refresh] [--really-refresh] [--cacheinfo] [--chmod=(+|-)x] [--assume-unchanged] [--info-only] [--force-remove] [--stdin] [--index-info] [--unresolve] [--again | -g] [--ignore-missing] [-z] [--verbose] [--] <file>...";
+"git update-index [-q] [--add] [--replace] [--remove] [--unmerged] [--refresh] [--really-refresh] [--cacheinfo] [--chmod=(+|-)x] [--assume-unchanged] [--checkout|--no-checkout] [--info-only] [--force-remove] [--stdin] [--index-info] [--unresolve] [--again | -g] [--ignore-missing] [-z] [--verbose] [--] <file>...";
 
 static unsigned char head_sha1[20];
 static unsigned char merge_head_sha1[20];
@@ -652,6 +658,14 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)
 				mark_valid_only = UNMARK_FLAG;
 				continue;
 			}
+			if (!strcmp(path, "--checkout")) {
+				mark_no_checkout_only = UNMARK_FLAG;
+				continue;
+			}
+			if (!strcmp(path, "--no-checkout")) {
+				mark_no_checkout_only = MARK_FLAG;
+				continue;
+			}
 			if (!strcmp(path, "--info-only")) {
 				info_only = 1;
 				continue;
diff --git a/t/t2104-update-index-no-checkout.sh b/t/t2104-update-index-no-checkout.sh
new file mode 100755
index 0000000..be9f913
--- /dev/null
+++ b/t/t2104-update-index-no-checkout.sh
@@ -0,0 +1,36 @@
+#!/bin/sh
+#
+# Copyright (c) 2008 Nguyễn Thái Ngọc Duy
+#
+
+test_description='git update-index no-checkout bits (a.k.a sparse checkout)'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+	mkdir sub &&
+	touch 1 2 sub/1 sub/2 &&
+	git add 1 2 sub/1 sub/2
+'
+
+test_expect_success 'index is at version 2' '
+	test "$(test-index-version < .git/index)" = 2
+'
+
+test_expect_success 'update-index --no-checkout' '
+	git update-index --no-checkout 1 sub/1 &&
+	test -z "$(git ls-files --sparse|grep 1)"'
+
+test_expect_success 'index is at version 3 after having some no-checkout entries' '
+	test "$(test-index-version < .git/index)" = 3
+'
+
+test_expect_success 'update-index --checkout' '
+	git update-index --checkout 1 sub/1 &&
+	test "$(git ls-files)" = "$(git ls-files --sparse)"'
+
+test_expect_success 'index version is back to 2 when there is no no-checkout entry' '
+	test "$(test-index-version < .git/index)" = 2
+'
+
+test_done
diff --git a/test-index-version.c b/test-index-version.c
new file mode 100644
index 0000000..bfaad9e
--- /dev/null
+++ b/test-index-version.c
@@ -0,0 +1,14 @@
+#include "cache.h"
+
+int main(int argc, const char **argv)
+{
+	struct cache_header hdr;
+	int version;
+
+	memset(&hdr,0,sizeof(hdr));
+	if (read(0, &hdr, sizeof(hdr)) != sizeof(hdr))
+		return 0;
+	version = ntohl(hdr.hdr_version);
+	printf("%d\n", version);
+	return 0;
+}
-- 
1.6.0.96.g2fad1.dirty

[PATCH 06/14] ls-files: Add tests for --sparse and friends

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 t/t3004-ls-files-sparse.sh            |   40 +++++++++++++++++++++++++++++++++
 t/t3004/cached.expected               |    5 ++++
 t/t3004/deleted.expected              |    1 +
 t/t3004/everything.expected           |   10 ++++++++
 t/t3004/modified.expected             |    2 +
 t/t3004/no-checkout.expected          |    2 +
 t/t3004/orphaned-no-checkout.expected |    3 ++
 t/t3004/orphaned.expected             |    1 +
 t/t3004/others.expected               |    2 +
 t/t3004/sparse-cached.expected        |    3 ++
 t/t3004/sparse-everything.expected    |   11 +++++++++
 11 files changed, 80 insertions(+), 0 deletions(-)
 create mode 100755 t/t3004-ls-files-sparse.sh
 create mode 100644 t/t3004/cached.expected
 create mode 100644 t/t3004/deleted.expected
 create mode 100644 t/t3004/everything.expected
 create mode 100644 t/t3004/modified.expected
 create mode 100644 t/t3004/no-checkout.expected
 create mode 100644 t/t3004/orphaned-no-checkout.expected
 create mode 100644 t/t3004/orphaned.expected
 create mode 100644 t/t3004/others.expected
 create mode 100644 t/t3004/sparse-cached.expected
 create mode 100644 t/t3004/sparse-everything.expected

diff --git a/t/t3004-ls-files-sparse.sh b/t/t3004-ls-files-sparse.sh
new file mode 100755
index 0000000..ec2c869
--- /dev/null
+++ b/t/t3004-ls-files-sparse.sh
@@ -0,0 +1,40 @@
+#!/bin/sh
+
+test_description="test ls-files in --sparse mode"
+
+. test-lib.sh
+
+test_ls_files() {
+	T=../t3004/$1.expected
+	shift
+	test_expect_success "ls-files $*" "git ls-files $* > result && test_cmp $T result"
+}
+
+test_expect_success 'setup' '
+	touch other orphaned no-checkout cached modified deleted &&
+	git add orphaned no-checkout cached modified deleted &&
+	git update-index --no-checkout orphaned no-checkout &&
+	echo modified >> modified &&
+	rm no-checkout deleted
+'
+
+test_ls_files cached
+test_ls_files cached --cached
+test_ls_files sparse-cached --sparse
+test_ls_files sparse-cached --sparse --cached
+test_ls_files no-checkout --no-checkout
+test_ls_files no-checkout --sparse --no-checkout
+test_ls_files orphaned --orphaned
+test_ls_files orphaned --sparse --orphaned
+test_ls_files orphaned-no-checkout -v --no-checkout --orphaned
+test_ls_files orphaned-no-checkout -v --sparse --no-checkout --orphaned
+test_ls_files deleted --deleted
+test_ls_files deleted --sparse --deleted
+test_ls_files modified --modified
+test_ls_files modified --sparse --modified
+test_ls_files others --others
+test_ls_files others --sparse --others
+test_ls_files everything -v --cached --deleted --modified --others
+test_ls_files sparse-everything -v --cached --no-checkout --orphaned --deleted --modified --others
+
+test_done
\ No newline at end of file
diff --git a/t/t3004/cached.expected b/t/t3004/cached.expected
new file mode 100644
index 0000000..6fd0c78
--- /dev/null
+++ b/t/t3004/cached.expected
@@ -0,0 +1,5 @@
+cached
+deleted
+modified
+no-checkout
+orphaned
diff --git a/t/t3004/deleted.expected b/t/t3004/deleted.expected
new file mode 100644
index 0000000..71779d2
--- /dev/null
+++ b/t/t3004/deleted.expected
@@ -0,0 +1 @@
+deleted
diff --git a/t/t3004/everything.expected b/t/t3004/everything.expected
new file mode 100644
index 0000000..6000328
--- /dev/null
+++ b/t/t3004/everything.expected
@@ -0,0 +1,10 @@
+? other
+? result
+H cached
+H deleted
+H modified
+H no-checkout
+H orphaned
+R deleted
+C deleted
+C modified
diff --git a/t/t3004/modified.expected b/t/t3004/modified.expected
new file mode 100644
index 0000000..644a96e
--- /dev/null
+++ b/t/t3004/modified.expected
@@ -0,0 +1,2 @@
+deleted
+modified
diff --git a/t/t3004/no-checkout.expected b/t/t3004/no-checkout.expected
new file mode 100644
index 0000000..b2a429f
--- /dev/null
+++ b/t/t3004/no-checkout.expected
@@ -0,0 +1,2 @@
+no-checkout
+orphaned
diff --git a/t/t3004/orphaned-no-checkout.expected b/t/t3004/orphaned-no-checkout.expected
new file mode 100644
index 0000000..d687ef0
--- /dev/null
+++ b/t/t3004/orphaned-no-checkout.expected
@@ -0,0 +1,3 @@
+- no-checkout
+- orphaned
+O orphaned
diff --git a/t/t3004/orphaned.expected b/t/t3004/orphaned.expected
new file mode 100644
index 0000000..571b267
--- /dev/null
+++ b/t/t3004/orphaned.expected
@@ -0,0 +1 @@
+orphaned
diff --git a/t/t3004/others.expected b/t/t3004/others.expected
new file mode 100644
index 0000000..bf5bf2b
--- /dev/null
+++ b/t/t3004/others.expected
@@ -0,0 +1,2 @@
+other
+result
diff --git a/t/t3004/sparse-cached.expected b/t/t3004/sparse-cached.expected
new file mode 100644
index 0000000..3453483
--- /dev/null
+++ b/t/t3004/sparse-cached.expected
@@ -0,0 +1,3 @@
+cached
+deleted
+modified
diff --git a/t/t3004/sparse-everything.expected b/t/t3004/sparse-everything.expected
new file mode 100644
index 0000000..5df0599
--- /dev/null
+++ b/t/t3004/sparse-everything.expected
@@ -0,0 +1,11 @@
+? other
+? result
+H cached
+H deleted
+H modified
+- no-checkout
+- orphaned
+O orphaned
+R deleted
+C deleted
+C modified
-- 
1.6.0.96.g2fad1.dirty

[PATCH 07/14] Prevent diff machinery from examining worktree outside sparse checkout

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 diff-lib.c |    5 +++--
 diff.c     |    4 +++-
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/diff-lib.c b/diff-lib.c
index ae96c64..992280b 100644
--- a/diff-lib.c
+++ b/diff-lib.c
@@ -161,7 +161,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)
 				continue;
 		}
 
-		if (ce_uptodate(ce))
+		if (ce_uptodate(ce) || ce_no_checkout(ce))
 			continue;
 
 		changed = check_removed(ce, &st);
@@ -348,6 +348,8 @@ static void do_oneway_diff(struct unpack_trees_options *o,
 	struct rev_info *revs = cbdata->revs;
 	int match_missing, cached;
 
+	/* if the entry is not checked out, don't examine work tree */
+	cached = o->index_only || (idx && ce_no_checkout(idx));
 	/*
 	 * Backward compatibility wart - "diff-index -m" does
 	 * not mean "do not ignore merges", but "match_missing".
@@ -355,7 +357,6 @@ static void do_oneway_diff(struct unpack_trees_options *o,
 	 * But with the revision flag parsing, that's found in
 	 * "!revs->ignore_merges".
 	 */
-	cached = o->index_only;
 	match_missing = !revs->ignore_merges;
 
 	if (cached && idx && ce_stage(idx)) {
diff --git a/diff.c b/diff.c
index a2dd931..b5b7249 100644
--- a/diff.c
+++ b/diff.c
@@ -1793,8 +1793,10 @@ static int reuse_worktree_file(const char *name, const unsigned char *sha1, int
 
 	/*
 	 * If ce matches the file in the work tree, we can reuse it.
+	 * For sparse checkout case, ce_uptodate() may be true although
+	 * the file may or may not exist in the work tree.
 	 */
-	if (ce_uptodate(ce) ||
+	if ((ce_uptodate(ce) && ce_checkout(ce)) ||
 	    (!lstat(name, &st) && !ce_match_stat(ce, &st, 0)))
 		return 1;
 
-- 
1.6.0.96.g2fad1.dirty

[PATCH 08/14] checkout_entry(): CE_NO_CHECKOUT on checked out entries.

[Nguyễn Thái Ngọc Duy]

With this you can just do "git checkout some-files" to
widen your checkout. One caveat though: caller must save
the index.

For all of its callers (unpack_trees(), checkout-index, checkout
and apply), only "git apply" does not write index back.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 entry.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/entry.c b/entry.c
index aa2ee46..305f8d3 100644
--- a/entry.c
+++ b/entry.c
@@ -230,5 +230,6 @@ int checkout_entry(struct cache_entry *ce, const struct checkout *state, char *t
 	} else if (state->not_new)
 		return 0;
 	create_directories(path, state);
+	ce_mark_checkout(ce);
 	return write_entry(ce, path, state, 0);
 }
-- 
1.6.0.96.g2fad1.dirty

[PATCH 09/14] grep: skip files outside sparse checkout area

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-grep.txt |    4 +++-
 builtin-grep.c             |    7 ++++++-
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt
index fa4d133..ee359c9 100644
--- a/Documentation/git-grep.txt
+++ b/Documentation/git-grep.txt
@@ -24,7 +24,9 @@ SYNOPSIS
 DESCRIPTION
 -----------
 Look for specified patterns in the working tree files, blobs
-registered in the index file, or given tree objects.
+registered in the index file, or given tree objects. By default
+it will search in the working tree files. When in sparse checkout
+mode, it only searches checked-out files.
 
 
 OPTIONS
diff --git a/builtin-grep.c b/builtin-grep.c
index 3a51662..d5507d7 100644
--- a/builtin-grep.c
+++ b/builtin-grep.c
@@ -343,6 +343,8 @@ static int external_grep(struct grep_opt *opt, const char **paths, int cached)
 			continue;
 		if (!pathspec_matches(paths, ce->name))
 			continue;
+		if (ce_no_checkout(ce))
+			continue;
 		name = ce->name;
 		if (name[0] == '-') {
 			int len = ce_namelen(ce);
@@ -404,8 +406,11 @@ static int grep_cache(struct grep_opt *opt, const char **paths, int cached)
 				continue;
 			hit |= grep_sha1(opt, ce->sha1, ce->name, 0);
 		}
-		else
+		else {
+			if (ce_no_checkout(ce))
+				continue;
 			hit |= grep_file(opt, ce->name);
+		}
 		if (ce_stage(ce)) {
 			do {
 				nr++;
-- 
1.6.0.96.g2fad1.dirty

[PATCH 10/14] ls-files: support "sparse patterns", used to form sparse checkout areas

[Nguyễn Thái Ngọc Duy]

This implements sparse patterns and adds --narrow-match option in order
to test the patterns.

Sparse patterns are basically like .gitignore patterns, but they can be
combined in one line, separating by colons like $PATH.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-checkout.txt   |   45 +++++++++++++
 Documentation/git-ls-files.txt   |    8 ++-
 builtin-ls-files.c               |   21 +++++-
 t/t3003-ls-files-narrow-match.sh |   39 +++++++++++
 t/t3003/1                        |    3 +
 t/t3003/12                       |    6 ++
 t/t3003/clone-escape             |    4 +
 t/t3003/cur-12                   |    2 +
 t/t3003/root-sub-1               |    1 +
 t/t3003/slash-1                  |    1 +
 t/t3003/sub-1                    |    2 +
 t/t3003/sub-only                 |    3 +
 t/t3003/subsub-slash             |    3 +
 unpack-trees.c                   |  136 ++++++++++++++++++++++++++++++++++++++
 unpack-trees.h                   |   18 +++++
 15 files changed, 288 insertions(+), 4 deletions(-)
 create mode 100755 t/t3003-ls-files-narrow-match.sh
 create mode 100644 t/t3003/1
 create mode 100644 t/t3003/12
 create mode 100644 t/t3003/clone-escape
 create mode 100644 t/t3003/cur-12
 create mode 100644 t/t3003/root-sub-1
 create mode 100644 t/t3003/slash-1
 create mode 100644 t/t3003/sub
 create mode 100644 t/t3003/sub-1
 create mode 100644 t/t3003/sub-only
 create mode 100644 t/t3003/subsub-slash

diff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt
index 2b344e1..d6f94a6 100644
--- a/Documentation/git-checkout.txt
+++ b/Documentation/git-checkout.txt
@@ -205,6 +205,51 @@ is "assume unchanged" bit just ignores corresponding files in working
 directory while sparse checkout goes a bit farther, remove those files
 when it is safe to do so.
 
+Sparse patterns
+---------------
+
+Sparse patterns specify how do you want to form your checkout area.
+Many patterns can be specified on one line, separated by colons.
+The patterns specify what files should or should not be checked out
+on working directory (depends on the option used with the patterns).
+Patterns have the following format:
+
+ - An optional prefix '!' which negates the pattern; any
+   matching file by a previous pattern will become
+   unmatched again.  If a negated pattern matches, this will
+   override lower precedence patterns sources.
+
+ - If the pattern ends with a slash, it is removed for the
+   purpose of the following description, but it would only find
+   a match with a directory.  In other words, `foo/` will match a
+   directory `foo` and paths underneath it, but will not match a
+   regular file or a symbolic link `foo` (this is consistent
+   with the way how pathspec works in general in git).
+
+ - If the pattern does not contain a slash '/', git treats it as
+   a shell glob pattern and checks for a match against the
+   pathname without leading directories.
+
+ - Otherwise, git treats the pattern as a shell glob suitable
+   for consumption by fnmatch(3) with the FNM_PATHNAME flag:
+   wildcards in the pattern will not match a / in the pathname.
+   For example, "Documentation/\*.html" matches
+   "Documentation/git.html" but not
+   "Documentation/ppc/ppc.html".  A leading slash matches the
+   beginning of the pathname; for example, "/*.c" matches
+   "cat-file.c" but not "mozilla-sha1/sha1.c".
+
+ - Patterns begin with a slash will match against full pathname,
+   as opposed to normal case when it only matches pathnames relative
+   to current working directory.
+
+ - Patterns begin with "./" are treated like normal patterns. That is
+   it will follow above rules. But since it has a slash inside,
+   "fnmatch rule" will apply. This is a work-around when you do not
+   want to apply "no slash" rule.
+
+ - Because colons are used to separate patterns, you cannot put them
+   in patterns directly. You must quote them using backslash.
 
 EXAMPLES
 --------
diff --git a/Documentation/git-ls-files.txt b/Documentation/git-ls-files.txt
index 1de68e2..fbed73b 100644
--- a/Documentation/git-ls-files.txt
+++ b/Documentation/git-ls-files.txt
@@ -12,7 +12,7 @@ SYNOPSIS
 'git ls-files' [-z] [-t] [-v]
 		(--[cached|deleted|others|ignored|stage|unmerged|killed|modified|orphaned|no-checkout])\*
 		(-[c|d|o|i|s|u|k|m])\*
-		[--sparse]
+		[--sparse] [--narrow-match=<sparse patterns>]
 		[-x <pattern>|--exclude=<pattern>]
 		[-X <file>|--exclude-from=<file>]
 		[--exclude-per-directory=<file>]
@@ -90,6 +90,12 @@ OPTIONS
 	No-checkout entries can be shown using --orphaned or
 	--no-checkout (or both).
 
+--narrow-match=<sparse patterns>::
+	This option can be used to test sparse patterns. The given sparse patterns will
+	be used to filter ls-files output. Entries not matching the spec will be
+	ignored. This option can only be used with --cached or --stage.
+	See linkgit:git-checkout[1] for more information about sparse patterns.
+
 -z::
 	\0 line termination on output.
 
diff --git a/builtin-ls-files.c b/builtin-ls-files.c
index 873de15..1c81022 100644
--- a/builtin-ls-files.c
+++ b/builtin-ls-files.c
@@ -10,6 +10,8 @@
 #include "dir.h"
 #include "builtin.h"
 #include "tree.h"
+#include "tree-walk.h"
+#include "unpack-trees.h"
 
 static int abbrev;
 static int show_deleted;
@@ -31,6 +33,7 @@ static const char **pathspec;
 static int error_unmatch;
 static char *ps_matched;
 static const char *with_tree;
+static struct narrow_spec *narrow_spec;
 
 static const char *tag_cached = "";
 static const char *tag_unmerged = "";
@@ -187,7 +190,7 @@ static void show_ce_entry(const char *tag, struct cache_entry *ce)
 	int len = prefix_len;
 	int offset = prefix_offset;
 
-	if (len >= ce_namelen(ce))
+	if (len >= ce_namelen(ce) && !narrow_spec)
 		die("git ls-files: internal error - cache entry not superset of prefix");
 
 	if (pathspec && !pathspec_match(pathspec, ps_matched, ce->name, len))
@@ -260,6 +263,8 @@ static void show_files(struct dir_struct *dir, const char *prefix)
 			}
 			if (!(show_cached | show_stage))
 				continue;
+			if (narrow_spec && !match_narrow_spec(narrow_spec, ce->name))
+				continue;
 			show_ce_entry(ce_stage(ce) ? tag_unmerged : tag_cached, ce);
 		}
 	}
@@ -439,7 +444,7 @@ int report_path_error(const char *ps_matched, const char **pathspec, int prefix_
 
 static const char ls_files_usage[] =
 	"git ls-files [-z] [-t] [-v] (--[cached|deleted|others|stage|unmerged|killed|modified|orphaned|no-checkout])* "
-	"[ --sparse ] "
+	"[ --sparse ] [--narrow-match=<narrow_spec>] "
 	"[ --ignored ] [--exclude=<pattern>] [--exclude-from=<file>] "
 	"[ --exclude-per-directory=<filename> ] [--exclude-standard] "
 	"[--full-name] [--abbrev] [--] [<file>]*";
@@ -498,6 +503,10 @@ int cmd_ls_files(int argc, const char **argv, const char *prefix)
 			sparse_checkout = 1;
 			continue;
 		}
+		if (!prefixcmp(arg, "--narrow-match=")) {
+			narrow_spec = parse_narrow_spec(arg+15, prefix);
+			continue;
+		}
 		if (!strcmp(arg, "-d") || !strcmp(arg, "--deleted")) {
 			show_deleted = 1;
 			continue;
@@ -629,8 +638,14 @@ int cmd_ls_files(int argc, const char **argv, const char *prefix)
 	      show_killed | show_modified | show_orphaned | show_no_checkout))
 		show_cached = 1;
 
+	if (narrow_spec && !show_cached && !show_stage)
+		die("ls-files: --narrow-match can only be used with either --cached or --stage");
+
+	if (narrow_spec && narrow_spec->has_root && prefix_offset != 0)
+		die("ls-files: --narrow-match with root matching patterns requires --full-name");
+
 	read_cache();
-	if (prefix)
+	if (prefix && (!narrow_spec || !narrow_spec->has_root))
 		prune_cache(prefix);
 	if (with_tree) {
 		/*
diff --git a/t/t3003-ls-files-narrow-match.sh b/t/t3003-ls-files-narrow-match.sh
new file mode 100755
index 0000000..5611cab
--- /dev/null
+++ b/t/t3003-ls-files-narrow-match.sh
@@ -0,0 +1,39 @@
+#!/bin/sh
+
+test_description='This test is for narrow spec matching'
+
+. test-lib.sh
+
+D="$(cd ..;pwd)"/t3003
+
+test_pattern() {
+	test_expect_success "pattern $1" '
+		(
+		if [ -n "'$3'" ]; then cd '$3'; fi
+		git ls-files --full-name --narrow-match="'"$2"'" > result &&
+		diff -u result "'"$D/$1"'"
+		)
+	'
+}
+
+test_expect_success 'setup' '
+	touch 1 2 3 "1:2" &&
+	mkdir -p sub/subsub &&
+	touch sub/1 sub/2 sub/3 &&
+	touch sub/subsub/1 sub/subsub/2 sub/subsub/3 &&
+	git add .
+'
+
+test_pattern 1 1
+test_pattern sub sub
+test_pattern sub-1 1 sub
+test_pattern root-sub-1 /1 sub
+test_pattern subsub-slash subsub/ sub
+test_pattern sub-only 'sub/:!sub/subsub/'
+test_pattern 12 1:2
+test_pattern cur-12 ./1:./2
+test_pattern slash-1 'sub/*1'
+test_pattern clone-escape '1\:2:1'
+
+test_done
+
diff --git a/t/t3003/1 b/t/t3003/1
new file mode 100644
index 0000000..9b73321
--- /dev/null
+++ b/t/t3003/1
@@ -0,0 +1,3 @@
+1
+sub/1
+sub/subsub/1
diff --git a/t/t3003/12 b/t/t3003/12
new file mode 100644
index 0000000..5d71811
--- /dev/null
+++ b/t/t3003/12
@@ -0,0 +1,6 @@
+1
+2
+sub/1
+sub/2
+sub/subsub/1
+sub/subsub/2
diff --git a/t/t3003/clone-escape b/t/t3003/clone-escape
new file mode 100644
index 0000000..11cdf68
--- /dev/null
+++ b/t/t3003/clone-escape
@@ -0,0 +1,4 @@
+1
+1:2
+sub/1
+sub/subsub/1
diff --git a/t/t3003/cur-12 b/t/t3003/cur-12
new file mode 100644
index 0000000..1191247
--- /dev/null
+++ b/t/t3003/cur-12
@@ -0,0 +1,2 @@
+1
+2
diff --git a/t/t3003/root-sub-1 b/t/t3003/root-sub-1
new file mode 100644
index 0000000..d00491f
--- /dev/null
+++ b/t/t3003/root-sub-1
@@ -0,0 +1 @@
+1
diff --git a/t/t3003/slash-1 b/t/t3003/slash-1
new file mode 100644
index 0000000..5798e42
--- /dev/null
+++ b/t/t3003/slash-1
@@ -0,0 +1 @@
+sub/1
diff --git a/t/t3003/sub b/t/t3003/sub
new file mode 100644
index 0000000..e69de29
diff --git a/t/t3003/sub-1 b/t/t3003/sub-1
new file mode 100644
index 0000000..3ef951a
--- /dev/null
+++ b/t/t3003/sub-1
@@ -0,0 +1,2 @@
+sub/1
+sub/subsub/1
diff --git a/t/t3003/sub-only b/t/t3003/sub-only
new file mode 100644
index 0000000..3115212
--- /dev/null
+++ b/t/t3003/sub-only
@@ -0,0 +1,3 @@
+sub/1
+sub/2
+sub/3
diff --git a/t/t3003/subsub-slash b/t/t3003/subsub-slash
new file mode 100644
index 0000000..bc585b0
--- /dev/null
+++ b/t/t3003/subsub-slash
@@ -0,0 +1,3 @@
+sub/subsub/1
+sub/subsub/2
+sub/subsub/3
diff --git a/unpack-trees.c b/unpack-trees.c
index e59d144..ce4c826 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -726,6 +726,142 @@ static void show_stage_entry(FILE *o,
 }
 #endif
 
+struct narrow_spec *parse_narrow_spec(const char *spec, const char *prefix)
+{
+	struct narrow_spec *ns;
+	struct narrow_pattern *p;
+	const char *start = spec, *end;
+	int has_wildcards, has_slashes;
+
+	ns = xmalloc(sizeof(*ns));
+	memset(ns, 0, sizeof(*ns));
+	if (prefix)
+		ns->prefix = xstrdup(prefix);
+
+	while (*start) {
+		end = start;
+		has_slashes = has_wildcards = 0;
+		while (*end && *end != ':') {
+			if (*end == '*' || *end == '[' || *end == '?')
+				has_wildcards = 1;
+			if (*end == '/')
+				has_slashes = 1;
+			if (*end == '\\') {
+				end++;
+				has_wildcards = 1;
+				if (*end == '\0') /* trailing backslash */
+					break;
+			}
+			end++;
+		}
+		if (start == end)
+			continue;
+
+		p = xmalloc(offsetof(struct narrow_pattern, pattern)+(end-start)+1);
+		p->negative = *start == '!';
+		if (p->negative)
+			start++;
+		p->has_slashes = has_slashes;
+		p->has_wildcards = has_wildcards;
+		p->has_trailing_slash = end[-1] == '/';
+		p->has_root = *start == '/';
+		if (p->has_root)
+			start++;
+		else if (*start == '.' && start[1] == '/')
+			start += 2;
+		p->len = end-start;
+		memcpy(p->pattern, start, p->len);
+		p->pattern[p->len] = '\0';
+
+		ALLOC_GROW(ns->patterns, ns->nr + 1, ns->alloc);
+		ns->patterns[ns->nr++] = p;
+		ns->has_root |= p->has_root;
+
+		if (*end != ':')
+			break;
+		start = end + 1;
+	}
+	return ns;
+}
+
+int match_narrow_spec(struct narrow_spec *spec, const char *path)
+{
+	int i, prefix_len = 0;
+
+	if (!spec || !spec->nr)
+		return 1; /* always match if spec is NULL */
+
+	if (spec->prefix) {
+		/*
+		 * optimization:
+		 * if there is no pattern with leading slash
+		 * then it is safe to only match inside prefix
+		 */
+		if (!spec->has_root && prefixcmp(path, spec->prefix))
+			return 0;
+		prefix_len = strlen(spec->prefix);
+	}
+
+	for (i = spec->nr - 1;i >= 0; i--) {
+		struct narrow_pattern *p = spec->patterns[i];
+		const char *new_path = path + prefix_len;
+		int match;
+
+		if (p->has_root)
+			new_path = path; /* match full path */
+		else if (spec->has_root) {
+			if (prefixcmp(path, spec->prefix))
+				continue;
+		}
+		/* !spec->has_root case has been handled above */
+
+		if (p->has_trailing_slash) {
+			/* the only "wildcard" here is backslash escape */
+			if (p->has_wildcards) {
+				char *unescaped_pattern = xstrdup(p->pattern);
+				char *src, *dst;
+
+				src = dst = unescaped_pattern;
+				while (*src) {
+					if (*src == '\\')
+						src++;
+					if (src != dst)
+						*dst = *src;
+					src++;
+					dst++;
+				}
+				*dst = '\0';
+				match = prefixcmp(new_path, unescaped_pattern) == 0;
+				free(unescaped_pattern);
+			}
+			else
+				match = prefixcmp(new_path, p->pattern) == 0;
+		}
+		else if (p->has_slashes) {
+			if (p->has_wildcards)
+				match = fnmatch(p->pattern, new_path, FNM_PATHNAME) == 0;
+			else
+				match = strcmp(p->pattern, new_path) == 0;
+		}
+		else {
+			const char *basename = strrchr(path + prefix_len, '/');
+			if (basename)
+				basename++;
+			else
+				basename = path + prefix_len;
+			if (p->has_wildcards)
+				match = fnmatch(p->pattern, basename, 0) == 0;
+			else
+				match = strcmp(p->pattern, basename) == 0;
+		}
+		if (match)
+			return p->negative ? 0 : 1;
+	}
+
+	/* no pattern is matched */
+	return 0;
+}
+
 int threeway_merge(struct cache_entry **stages, struct unpack_trees_options *o)
 {
 	struct cache_entry *index;
diff --git a/unpack-trees.h b/unpack-trees.h
index 0d26f3d..6b1971f 100644
--- a/unpack-trees.h
+++ b/unpack-trees.h
@@ -16,6 +16,22 @@ struct unpack_trees_error_msgs {
 	const char *bind_overlap;
 };
 
+struct narrow_spec {
+	int nr;
+	int alloc;
+	int has_root:1;
+	const char *prefix;
+	struct narrow_pattern {
+		int len;
+		int has_root:1;
+		int has_slashes:1;
+		int has_wildcards:1;
+		int has_trailing_slash:1;
+		int negative:1;
+		char pattern[FLEX_ARRAY];
+	} **patterns;
+};
+
 struct unpack_trees_options {
 	unsigned int reset:1,
 		     merge:1,
@@ -48,6 +64,8 @@ struct unpack_trees_options {
 extern int unpack_trees(unsigned n, struct tree_desc *t,
 		struct unpack_trees_options *options);
 
+struct narrow_spec *parse_narrow_spec(const char *spec, const char *prefix);
+int match_narrow_spec(struct narrow_spec *spec, const char *path);
 int threeway_merge(struct cache_entry **stages, struct unpack_trees_options *o);
 int twoway_merge(struct cache_entry **src, struct unpack_trees_options *o);
 int bind_merge(struct cache_entry **src, struct unpack_trees_options *o);
-- 
1.6.0.96.g2fad1.dirty

[PATCH 11/14] unpack_trees(): add support for sparse checkout

[Nguyễn Thái Ngọc Duy]

This patch teaches unpack_trees() to checkout/remove entries
on working directories appropriately when sparse checkout area is
changed. There are three kind of changes:

 - new_narrow_path: reset workdir to a completely new checkout area
 - add_narrow_path: keep current areas and add more entries
 - remove_narrow_path: remove some entries from current areas

CE_WD_REMOVE is introduced to remove entries from working directories,
but still keep them in index

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 cache.h        |    3 ++
 unpack-trees.c |   74 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 unpack-trees.h |    4 +++
 3 files changed, 80 insertions(+), 1 deletions(-)

diff --git a/cache.h b/cache.h
index 6e875d5..f4025b5 100644
--- a/cache.h
+++ b/cache.h
@@ -167,6 +167,9 @@ struct cache_entry {
 #define CE_HASHED    (0x100000)
 #define CE_UNHASHED  (0x200000)
 
+/* Only remove in work directory, not index */
+#define CE_WD_REMOVE (0x400000)
+
 /*
  * Extended on-disk flags
  */
diff --git a/unpack-trees.c b/unpack-trees.c
index ce4c826..10f377c 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -96,7 +96,7 @@ static int check_updates(struct unpack_trees_options *o)
 	if (o->update && o->verbose_update) {
 		for (total = cnt = 0; cnt < index->cache_nr; cnt++) {
 			struct cache_entry *ce = index->cache[cnt];
-			if (ce->ce_flags & (CE_UPDATE | CE_REMOVE))
+			if (ce->ce_flags & (CE_UPDATE | CE_REMOVE | CE_WD_REMOVE))
 				total++;
 		}
 
@@ -108,6 +108,13 @@ static int check_updates(struct unpack_trees_options *o)
 	for (i = 0; i < index->cache_nr; i++) {
 		struct cache_entry *ce = index->cache[i];
 
+		if (ce->ce_flags & CE_WD_REMOVE) {
+			display_progress(progress, ++cnt);
+			if (o->update)
+				unlink_entry(ce);
+			continue;
+		}
+
 		if (ce->ce_flags & CE_REMOVE) {
 			display_progress(progress, ++cnt);
 			if (o->update)
@@ -133,6 +140,66 @@ static int check_updates(struct unpack_trees_options *o)
 	return errs != 0;
 }
 
+static int verify_uptodate(struct cache_entry *ce, struct unpack_trees_options *o);
+static int apply_narrow_spec(struct unpack_trees_options *o)
+{
+	struct index_state *index = &o->result;
+	int i;
+
+	if (!(o->new_narrow_path | o->add_narrow_path | o->remove_narrow_path))
+		return 0;
+
+	for (i = 0; i < index->cache_nr; i++) {
+		struct cache_entry *ce = index->cache[i];
+		int was_checkout = ce_checkout(ce);
+		int match = match_narrow_spec(o->narrow_spec, ce->name);
+
+		if (ce_stage(ce))
+			continue;
+
+		if (o->new_narrow_path) {
+			if (match)
+				ce_mark_checkout(ce);
+			else
+				ce_mark_no_checkout(ce);
+		}
+
+		if (o->add_narrow_path && match)
+			ce_mark_checkout(ce);
+
+		if (o->remove_narrow_path && match)
+			ce_mark_no_checkout(ce);
+
+		/* Update worktree, add/remove entries if needed */
+
+		/*
+		 * We only care about files getting into the checkout area
+		 * If merge strategies want to remove some, go ahead
+		 */
+		if (ce->ce_flags & CE_REMOVE)
+			continue;
+
+		if (was_checkout && ce_no_checkout(ce)) {
+			/*
+			 * If CE_UPDATE is set, verify_uptodate() must be called already
+			 * also stat info may have lost after merged_entry() so calling
+			 * verify_uptodate() again may fail
+			 */
+			if (!(ce->ce_flags & CE_UPDATE) && verify_uptodate(ce, o))
+				return -1;
+			ce->ce_flags |= CE_WD_REMOVE;
+		}
+		if (!was_checkout && ce_checkout(ce))
+			ce->ce_flags |= CE_UPDATE;
+
+		/* merge strategies may set CE_UPDATE outside checkout area */
+		if (ce_no_checkout(ce))
+			ce->ce_flags &= ~CE_UPDATE;
+
+	}
+	return 0;
+}
+
 static inline int call_unpack_fn(struct cache_entry **src, struct unpack_trees_options *o)
 {
 	int ret = o->fn(src, o);
@@ -409,6 +476,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 	if (o->trivial_merges_only && o->nontrivial_merge)
 		return unpack_failed(o, "Merge requires file-level merging");
 
+	if (apply_narrow_spec(o))
+		return unpack_failed(o, NULL);
+
 	o->src_index = NULL;
 	ret = check_updates(o) ? (-2) : 0;
 	if (o->dst_index)
@@ -677,6 +747,8 @@ static int merged_entry(struct cache_entry *merge, struct cache_entry *old,
 				return -1;
 			invalidate_ce_path(old, o);
 		}
+		if (ce_no_checkout(old))
+			update |= CE_NO_CHECKOUT;
 	}
 	else {
 		if (verify_absent(merge, "overwritten", o))
diff --git a/unpack-trees.h b/unpack-trees.h
index 6b1971f..0d899b6 100644
--- a/unpack-trees.h
+++ b/unpack-trees.h
@@ -43,6 +43,9 @@ struct unpack_trees_options {
 		     aggressive:1,
 		     skip_unmerged:1,
 		     initial_checkout:1,
+		     new_narrow_path:1,
+		     add_narrow_path:2,
+		     remove_narrow_path:2,
 		     gently:1;
 	const char *prefix;
 	int pos;
@@ -54,6 +57,7 @@ struct unpack_trees_options {
 	int merge_size;
 
 	struct cache_entry *df_conflict_entry;
+	struct narrow_spec *narrow_spec;
 	void *unpack_data;
 
 	struct index_state *dst_index;
-- 
1.6.0.96.g2fad1.dirty

[PATCH 12/14] clone: support sparse checkout with --narrow-path option

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-clone.txt |   10 +++++++++-
 builtin-clone.c             |   13 +++++++++++++
 t/t5703-clone-narrow.sh     |   39 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 61 insertions(+), 1 deletions(-)
 create mode 100755 t/t5703-clone-narrow.sh

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 0e14e73..c283cf4 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -12,7 +12,8 @@ SYNOPSIS
 'git clone' [--template=<template_directory>]
 	  [-l] [-s] [--no-hardlinks] [-q] [-n] [--bare] [--mirror]
 	  [-o <name>] [-u <upload-pack>] [--reference <repository>]
-	  [--depth <depth>] [--] <repository> [<directory>]
+	  [--depth <depth>] [--narrow-path=<sparse patterns>] [--]
+	  <repository> [<directory>]
 
 DESCRIPTION
 -----------
@@ -94,6 +95,13 @@ then the cloned repository will become corrupt.
 -n::
 	No checkout of HEAD is performed after the clone is complete.
 
+--narrow-path=<sparse patterns>::
+	Make a sparse checkout instead of full one. The checkout area
+	will be narrowed to specific areas based on given sparse
+	patterns. This option will not work with either --no-checkout
+	or --bare.  Please refer to linkgit:git-checkout[1] for more
+	detail on sparse checkout and sparse patterns.
+
 --bare::
 	Make a 'bare' GIT repository.  That is, instead of
 	creating `<directory>` and placing the administrative
diff --git a/builtin-clone.c b/builtin-clone.c
index a4b8790..5ee8362 100644
--- a/builtin-clone.c
+++ b/builtin-clone.c
@@ -36,6 +36,7 @@ static const char * const builtin_clone_usage[] = {
 static int option_quiet, option_no_checkout, option_bare, option_mirror;
 static int option_local, option_no_hardlinks, option_shared;
 static char *option_template, *option_reference, *option_depth;
+static char *option_narrow_path;
 static char *option_origin = NULL;
 static char *option_upload_pack = "git-upload-pack";
 
@@ -43,6 +44,8 @@ static struct option builtin_clone_options[] = {
 	OPT__QUIET(&option_quiet),
 	OPT_BOOLEAN('n', "no-checkout", &option_no_checkout,
 		    "don't create a checkout"),
+	OPT_STRING(0, "narrow-path", &option_narrow_path, "prefixes",
+		    "limit checkout to specified areas (sparse checkout)"),
 	OPT_BOOLEAN(0, "bare", &option_bare, "create a bare repository"),
 	OPT_BOOLEAN(0, "naked", &option_bare, "create a bare repository"),
 	OPT_BOOLEAN(0, "mirror", &option_mirror,
@@ -378,10 +381,15 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 		if (option_origin)
 			die("--bare and --origin %s options are incompatible.",
 			    option_origin);
+		if (option_narrow_path)
+			die("--bare and --narrow-path options are incompatible.");
 		option_no_checkout = 1;
 		use_separate_remote = 0;
 	}
 
+	if (option_no_checkout && option_narrow_path)
+		die("--no-checkout and --narrow-path options are incompatible.");
+
 	if (!option_origin)
 		option_origin = "origin";
 
@@ -590,6 +598,11 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 		opts.src_index = &the_index;
 		opts.dst_index = &the_index;
 
+		if (option_narrow_path) {
+			opts.new_narrow_path = 1;
+			opts.narrow_spec = parse_narrow_spec(option_narrow_path, NULL);
+		}
+
 		tree = parse_tree_indirect(remote_head->old_sha1);
 		parse_tree(tree);
 		init_tree_desc(&t, tree->buffer, tree->size);
diff --git a/t/t5703-clone-narrow.sh b/t/t5703-clone-narrow.sh
new file mode 100755
index 0000000..66f9191
--- /dev/null
+++ b/t/t5703-clone-narrow.sh
@@ -0,0 +1,39 @@
+#!/bin/sh
+
+test_description='narrow clone'
+
+. ./test-lib.sh
+
+test_expect_success setup '
+	rm -fr .git &&
+	test_create_repo src &&
+	(
+		cd src
+		mkdir -p work/sub/dir
+		touch untracked tracked modified added
+		touch work/untracked work/tracked work/modified work/added
+		git add tracked work/tracked
+		git add modified work/modified
+		git commit -m initial
+	)
+
+'
+
+test_expect_success 'narrow clone incompatible with --bare' '
+	rm -fr dst &&
+	test_must_fail git clone --narrow-path=work --bare src dst
+'
+
+test_expect_success 'narrow clone incompatible with --no-checkout' '
+	rm -fr dst &&
+	test_must_fail git clone --narrow-path=work -n src dst
+'
+
+test_expect_success 'clone with --narrow-path' '
+	rm -fr dst &&
+	git clone --narrow-path=work src dst &&
+	cd dst &&
+	test -z "$(git ls-files --sparse | grep -v ^work/)"
+'
+
+test_done
-- 
1.6.0.96.g2fad1.dirty

[PATCH 13/14] checkout: add new options to support sparse checkout

[Nguyễn Thái Ngọc Duy]

This patch adds main interface to manipulate sparse checkout.
New options are added to support entering/updating/leaving sparse
checkout:

 --full: return to full checkout (default)
 --reset-path: set checkout area according to given spec
 --add-path/--remove-path: adjust current sparse checkout area

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-checkout.txt |   54 +++++++++++++++++++-
 builtin-checkout.c             |   37 ++++++++++++++
 t/t2011-checkout-sparse.sh     |  108 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 196 insertions(+), 3 deletions(-)
 create mode 100755 t/t2011-checkout-sparse.sh

diff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt
index d6f94a6..571e2a4 100644
--- a/Documentation/git-checkout.txt
+++ b/Documentation/git-checkout.txt
@@ -8,8 +8,10 @@ git-checkout - Checkout a branch or paths to the working tree
 SYNOPSIS
 --------
 [verse]
-'git checkout' [-q] [-f] [--track | --no-track] [-b <new_branch> [-l]] [-m] [<branch>]
-'git checkout' [-f|--ours|--theirs|-m|--conflict=<style>] [<tree-ish>] [--] <paths>...
+'git checkout' [-q] [-f] [--track | --no-track] [-b <new_branch> [-l]] [-m]
+	  [<sparse checkout options>] [<branch>]
+'git checkout' [-f|--ours|--theirs|-m|--conflict=<style>] [<tree-ish>]
+	  [<sparse checkout options>] [--] <paths>...
 
 DESCRIPTION
 -----------
@@ -34,6 +36,10 @@ used to specify a specific tree-ish (i.e. commit, tag or tree)
 to update the index for the given paths before updating the
 working tree.
 
+<sparse checkout options> include --full, --reset-path, --add-path
+and --remove-path. The last three require sparse patterns. Please refer
+to "sparse checkout" section for more information about this mode.
+
 The index may contain unmerged entries after a failed merge.  By
 default, if you try to check out such an entry from the index, the
 checkout operation will fail and nothing will be checked out.
@@ -117,6 +123,32 @@ should result in deletion of the path).
 When checking out paths from the index, this option lets you recreate
 the conflicted merge in the specified paths.
 
+--full::
+	Quit sparse checkout mode. Return to full checkout. This option
+	cannot be used with either --reset-path, --add-path,
+	--remove-path or <paths>.
+
+--reset-path=<sparse patterns>::
+	Re-apply new sparse patterns on current working directory to
+	form new checkout area. All no-checkout bits will be wiped
+	out before applying the patterns. This option cannot be used
+	with --full, --add-path, --remove-path or <paths>. Multiple
+	--reset-path is not allowed.
+
+--add-path=<sparse patterns>::
+	Checkout more areas specified by sparse patterns to current
+	checkout area. Already checked out entries are not affected.
+	This option cannot be used with --full, --reset-path,
+	--remove-path or <paths>. Multiple --add-path is not allowed.
+
+--remove-path=<sparse patterns>::
+	Narrow checkout area by removing files specified by sparse patterns
+	from current checkout area. This operation will fail if there
+	are unmerged or modified files in the removing areas. No-checkout
+	entries are not affected. This option cannot be used with --full,
+	--reset-path, --add-path or <paths>. Multiple --remove-path is not
+	allowed.
+
 --conflict=<style>::
 	The same as --merge option above, but changes the way the
 	conflicting hunks are presented, overriding the
@@ -186,7 +218,10 @@ Because sparse checkout uses a new index format, it will be
 incompatible with git prior to 1.6.0 regarding worktree operations.
 Operations that only need access to the repository itself, such as
 clone, push, or pull/fetch from another (normal) repository... should
-not be affected by sparse checkout.
+not be affected by sparse checkout. In order to make your working
+directory work again with those versions, you can use
+`git checkout --full` to return to normal mode (and compatible index
+format).
 
 In sparse checkout mode, checkout status of every files in your
 working directory will be recorded in index. If a file is marked
@@ -251,6 +286,19 @@ Patterns have the following format:
  - Because colons are used to separate patterns, you cannot put them
    in patterns directly. You must quote them using backslash.
 
+When you apply new sparse patterns to your working directory using either
+--reset-path, --add-path or --remove-path, it will update "checkout" status
+in index accordingly. Moreover, if a file is marked "no-checkout" and
+is present in working directory, it will be removed. If a file is
+turned from "no-checkout" to "checkout", then it will be added again
+to working directory. Modified and unmerged entries can't bear
+"no-checkout" status, if sparse patterns apply to them, "git checkout"
+will refuse to update working directory.
+
+Sparse patterns are not saved by "git checkout" anywhere in the repository.
+You can form your checkout area in one go with --reset-path option,
+or do it incrementally with --add-path and --remove-path.
+
 EXAMPLES
 --------
 
diff --git a/builtin-checkout.c b/builtin-checkout.c
index c7b0aad..d459bc6 100644
--- a/builtin-checkout.c
+++ b/builtin-checkout.c
@@ -33,6 +33,12 @@ struct checkout_opts {
 	const char *new_branch;
 	int new_branch_log;
 	enum branch_track track;
+
+	const char *prefix;
+	char *new_path;
+	char *add_path;
+	char *remove_path;
+	int all_path;
 };
 
 static int post_checkout_hook(struct commit *old, struct commit *new,
@@ -412,6 +418,23 @@ static int merge_working_tree(struct checkout_opts *opts,
 		tree = parse_tree_indirect(new->commit->object.sha1);
 		init_tree_desc(&trees[1], tree->buffer, tree->size);
 
+		if (opts->all_path) {
+			/* leave narrow_spec NULL */
+			topts.new_narrow_path = 1;
+		}
+		else if (opts->new_path) {
+			topts.narrow_spec = parse_narrow_spec(opts->new_path, opts->prefix);
+			topts.new_narrow_path = 1;
+		}
+		else if (opts->add_path) {
+			topts.narrow_spec = parse_narrow_spec(opts->add_path, opts->prefix);
+			topts.add_narrow_path = 1;
+		}
+		else if (opts->remove_path) {
+			topts.narrow_spec = parse_narrow_spec(opts->remove_path, opts->prefix);
+			topts.remove_narrow_path = 1;
+		}
+
 		ret = unpack_trees(2, trees, &topts);
 		if (ret == -1) {
 			/*
@@ -600,6 +623,10 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
 		OPT_BOOLEAN('m', "merge", &opts.merge, "merge"),
 		OPT_STRING(0, "conflict", &conflict_style, "style",
 			   "conflict style (merge or diff3)"),
+		OPT_BOOLEAN(0, "full", &opts.all_path, "quit sparse checkout"),
+		OPT_STRING(0, "reset-path", &opts.new_path, "prefixes", "reset to new sparse checkout"),
+		OPT_STRING(0, "add-path", &opts.add_path, "prefixes", "widen checkout area"),
+		OPT_STRING(0, "remove-path", &opts.remove_path, "prefixes", "narrow checkout area"),
 		OPT_END(),
 	};
 	int has_dash_dash;
@@ -610,6 +637,7 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
 	git_config(git_checkout_config, NULL);
 
 	opts.track = BRANCH_TRACK_UNSPECIFIED;
+	opts.prefix = prefix;
 
 	argc = parse_options(argc, argv, options, checkout_usage,
 			     PARSE_OPT_KEEP_DASHDASH);
@@ -639,6 +667,12 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
 	if (!opts.new_branch && (opts.track != git_branch_track))
 		die("git checkout: --track and --no-track require -b");
 
+	if (((opts.all_path ? 1 : 0) +
+	     (opts.new_path ? 1 : 0) +
+	     (opts.add_path ? 1 : 0) +
+	     (opts.remove_path ? 1 : 0)) > 1)
+		die("git checkout: --reset-path, --full, --add-path and --remove-path are incompatible");
+
 	if (opts.force && opts.merge)
 		die("git checkout: -f and -m are incompatible");
 
@@ -732,6 +766,9 @@ no_reference:
 		if (1 < !!opts.writeout_stage + !!opts.force + !!opts.merge)
 			die("git checkout: --ours/--theirs, --force and --merge are incompatible when\nchecking out of the index.");
 
+		if (opts.all_path || opts.new_path || opts.add_path || opts.remove_path)
+			die("git checkout: updating paths is incompatible with setting sparse checkout");
+
 		return checkout_paths(source_tree, pathspec, &opts);
 	}
 
diff --git a/t/t2011-checkout-sparse.sh b/t/t2011-checkout-sparse.sh
new file mode 100755
index 0000000..b5ccfe4
--- /dev/null
+++ b/t/t2011-checkout-sparse.sh
@@ -0,0 +1,108 @@
+#!/bin/sh
+
+test_description='sparse checkout'
+
+. ./test-lib.sh
+
+test_expect_success setup '
+	mkdir work1 work2 work3
+	touch one two three
+	touch work1/one work2/two work3/three
+	git add one work1/one
+	git commit -m work1
+	git add two work2/two
+	git commit -m work2
+	git add three work3/three
+	git commit -m work3
+'
+
+test_expect_success '--full on no-narrow checkout' '
+	git checkout --full
+'
+
+test_expect_success '--full and --reset-path incompatible' '
+	test_must_fail git checkout --full --reset-path=work1
+'
+
+test_expect_success 'limit worktree to work1 and work2' '
+	git checkout --reset-path=work1/:work2/ &&
+	test -f work1/one &&
+	test -f work2/two &&
+	! test -f work3/three
+'
+
+test_expect_success 'update worktree to work2 and work3' '
+	git checkout --reset-path=work2/:work3/ &&
+	! test -f work1/one &&
+	test -f work2/two &&
+	test -f work3/three
+'
+
+test_expect_success 'update narrow prefix with modification' '
+	echo modified >> work2/two &&
+	git checkout --reset-path=work1/:work2/ &&
+	test -f work1/one &&
+	test -f work2/two &&
+	! test -f work3/three &&
+	grep -q modified work2/two
+'
+
+test_expect_success 'update checkout should not lose modification' '
+	! git checkout --reset-path=work1/:work3/ &&
+	test -f work1/one &&
+	test -f work2/two &&
+	! test -f work3/three &&
+	grep -q modified work2/two
+'
+
+test_expect_success 'widen checkout area' '
+	git checkout --add-path=work3/ &&
+	test -f work1/one &&
+	test -f work2/two &&
+	test -f work3/three
+'
+
+test_expect_success 'narrow checkout area' '
+	git checkout --remove-path=work3/ &&
+	test -f work1/one &&
+	test -f work2/two &&
+	! test -f work3/three
+'
+
+test_expect_success 'update outside checkout area' '
+	echo one >> work1/one &&
+	git add work1/one &&
+	git commit -m update &&
+	git checkout --reset-path=work2/ &&
+	git checkout HEAD^ &&
+	git checkout master
+'
+
+test_expect_success 'conflict outside checkout area' '
+	git checkout --add-path=work1/one -b conflict HEAD~2 &&
+	echo two >> work1/one &&
+	git add work1/one &&
+	git commit -m conflict-update &&
+	git checkout --reset-path=work2/ master &&
+	test -z "$(git ls-files --sparse work1/one)"
+	git merge conflict
+	test $? = 1 &&
+	test -n "$(git ls-files --sparse work1/one)" &&
+	git reset --hard HEAD
+'
+
+test_expect_success 'removal outside checkout area' '
+	git rm work1/one &&
+	git commit -m remove &&
+	git checkout --reset-path=work2/ HEAD^
+'
+
+test_expect_success 'exit sparse checkout' '
+	git checkout --full &&
+	test -f work1/one &&
+	test -f work2/two &&
+	test -f work3/three &&
+	test one
+'
+
+test_done
-- 
1.6.0.96.g2fad1.dirty

[PATCH 14/14] wt-status: Show orphaned entries in "git status" output

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 wt-status.c |   39 +++++++++++++++++++++++++++++++++++++++
 wt-status.h |    1 +
 2 files changed, 40 insertions(+), 0 deletions(-)

diff --git a/wt-status.c b/wt-status.c
index 7cf890f..4c237e2 100644
--- a/wt-status.c
+++ b/wt-status.c
@@ -20,6 +20,7 @@ static char wt_status_colors[][COLOR_MAXLEN] = {
 	"\033[31m", /* WT_STATUS_CHANGED: red */
 	"\033[31m", /* WT_STATUS_UNTRACKED: red */
 	"\033[31m", /* WT_STATUS_NOBRANCH: red */
+	"\033[31m", /* WT_STATUS_ORPHANED: red */
 };
 
 enum untracked_status_type show_untracked_files = SHOW_NORMAL_UNTRACKED_FILES;
@@ -83,6 +84,16 @@ static void wt_status_print_dirty_header(struct wt_status *s,
 	color_fprintf_ln(s->fp, c, "#");
 }
 
+static void wt_status_print_orphaned_header(struct wt_status *s)
+{
+	const char *c = color(WT_STATUS_HEADER);
+	color_fprintf_ln(s->fp, c, "# Orphaned files:");
+	color_fprintf_ln(s->fp, c, "#   (these are tracked, but marked no-checkout and should not be present)");
+	color_fprintf_ln(s->fp, c, "#   (use \"git update-index --checkout\" to remove no-checkout status)");
+	color_fprintf_ln(s->fp, c, "#   (otherwise remove them to avoid confusion because git will ignore them)");
+	color_fprintf_ln(s->fp, c, "#");
+}
+
 static void wt_status_print_untracked_header(struct wt_status *s)
 {
 	const char *c = color(WT_STATUS_HEADER);
@@ -233,6 +244,33 @@ static void wt_status_print_changed(struct wt_status *s)
 	run_diff_files(&rev, 0);
 }
 
+static void wt_status_print_orphaned(struct wt_status *s)
+{
+	int i, show_header = 0;
+	struct stat st;
+	struct strbuf buf;
+
+	strbuf_init(&buf, 0);
+	for (i = 0; i < the_index.cache_nr; i++) {
+		struct cache_entry *ce = the_index.cache[i];
+
+		if (ce_checkout(ce) || lstat(ce->name, &st))
+			continue;
+
+		if (!show_header) {
+			wt_status_print_orphaned_header(s);
+			show_header = 1;
+		}
+
+		color_fprintf(s->fp, color(WT_STATUS_HEADER), "#\t");
+		color_fprintf_ln(s->fp, color(WT_STATUS_ORPHANED), "%s",
+				 quote_path(ce->name, -1, &buf, s->prefix));
+	}
+
+	if (show_header)
+		color_fprintf_ln(s->fp, color(WT_STATUS_HEADER),"#");
+}
+
 static void wt_status_print_submodule_summary(struct wt_status *s)
 {
 	struct child_process sm_summary;
@@ -373,6 +411,7 @@ void wt_status_print(struct wt_status *s)
 	}
 
 	wt_status_print_changed(s);
+	wt_status_print_orphaned(s);
 	if (wt_status_submodule_summary)
 		wt_status_print_submodule_summary(s);
 	if (show_untracked_files)
diff --git a/wt-status.h b/wt-status.h
index 78add09..52f1eb5 100644
--- a/wt-status.h
+++ b/wt-status.h
@@ -9,6 +9,7 @@ enum color_wt_status {
 	WT_STATUS_CHANGED,
 	WT_STATUS_UNTRACKED,
 	WT_STATUS_NOBRANCH,
+	WT_STATUS_ORPHANED,
 };
 
 enum untracked_status_type {
-- 
1.6.0.96.g2fad1.dirty

Re: [PATCH 01/14] Extend index to save more flags

[Jakub Narebski]

Comments below are just nitpicking. Feel free to diregard them...

Nguyễn Thái Ngọc Duy <pclouds@gmail.com> writes:

> The on-disk format of index only saves 16 bit flags, nearly all have
> been used. The last bit (CE_EXTENDED) is used to for future extension.
> 
> This patch extends index entry format to save more flags in future.
> The new entry format will be used when CE_EXTENDED bit is 1.
> 
> Because older implementation may not understand CE_EXTENDED bit and
> misread the new format, if there is any extended entry in index, index
> header version will turn 3, which makes it incompatible for older git.
> If there is none, header version will return to 2 again.
> 
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>

It would be nice if at least this part of series got accepted...

> ---
>  cache.h      |   58 ++++++++++++++++++++++++++++++++++++++++++++++++++++++----
>  read-cache.c |   51 +++++++++++++++++++++++++++++++++++++++++----------
>  2 files changed, 95 insertions(+), 14 deletions(-)
> 
> diff --git a/cache.h b/cache.h
> index f4b8ddf..77b6eb3 100644
> --- a/cache.h
> +++ b/cache.h
> @@ -109,6 +109,26 @@ struct ondisk_cache_entry {
>  	char name[FLEX_ARRAY]; /* more */
>  };
>  
> +/*
> + * This struct is used when CE_EXTENDED bit is 1
> + * The struct must match ondisk_cache_entry exactly from
> + * ctime till flags
> + */

Errr... "must match"? Wouldn't "does match" be better?
This type is defined below, not is to be defined...

> +struct ondisk_cache_entry_extended {
> +	struct cache_time ctime;
> +	struct cache_time mtime;
> +	unsigned int dev;
> +	unsigned int ino;
> +	unsigned int mode;
> +	unsigned int uid;
> +	unsigned int gid;
> +	unsigned int size;
> +	unsigned char sha1[20];
> +	unsigned short flags;
> +	unsigned short flags2;

flags and flags2? Why not flags1 and flags2, or flags[2], or flags and
flags_ext/flags_extended?

Just nitpicking.

> +	char name[FLEX_ARRAY]; /* more */
> +};
> +
>  struct cache_entry {
>  	unsigned int ce_ctime;
>  	unsigned int ce_mtime;
> @@ -130,7 +150,15 @@ struct cache_entry {
>  #define CE_VALID     (0x8000)
>  #define CE_STAGESHIFT 12
>  
> -/* In-memory only */
> +/*
> + * Range 0xFFFF0000 in ce_flags is divided into
> + * two parts: in-memory flags and on-disk ones.
> + * Flags in CE_EXTENDED_FLAGS will get saved on-disk

Semicolon at the end of below text to separate, I think. Or at least
comma.

> + * if you want to save a new flag, add it in
> + * CE_EXTENDED_FLAGS

Nice comment.

> + *
> + * In-memory only flags
> + */
>  #define CE_UPDATE    (0x10000)
>  #define CE_REMOVE    (0x20000)
>  #define CE_UPTODATE  (0x40000)
> @@ -140,6 +168,24 @@ struct cache_entry {
>  #define CE_UNHASHED  (0x200000)
>  
>  /*
> + * Extended on-disk flags
> + */
> +/* CE_EXTENDED2 is for future extension */
> +#define CE_EXTENDED2 0x80000000

Perhaps CE_RESERVED then?

> +
> +#define CE_EXTENDED_FLAGS (0)
> +
> +/*
> + * Safeguard to avoid saving wrong flags:
> + *  - CE_EXTENDED2 won't get saved until its semantic is known
> + *  - Bits in 0x0000FFFF have been saved in ce_flags already
> + *  - Bits in 0x003F0000 are currently in-memory flags
> + */
> +#if CE_EXTENDED_FLAGS & 0x80CFFFFF
> +#error "CE_EXTENDED_FLAGS out of range"
> +#endif

I don't quite understand the above fragment (especially with the fact
that CE_EXTENDED_FLAGS is defined as (0))...

> diff --git a/read-cache.c b/read-cache.c
> index c5a8659..667c36b 100644
> --- a/read-cache.c
> +++ b/read-cache.c
> @@ -1096,7 +1096,7 @@ static int verify_hdr(struct cache_header *hdr, unsigned long size)
>  
>  	if (hdr->hdr_signature != htonl(CACHE_SIGNATURE))
>  		return error("bad signature");
> -	if (hdr->hdr_version != htonl(2))
> +	if (hdr->hdr_version != htonl(2) && hdr->hdr_version != htonl(3))
>  		return error("bad index version");

By the way: what was index version 1?

[...]
-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: [PATCH 01/14] Extend index to save more flags

[Junio C Hamano]

Jakub Narebski <jnareb@gmail.com> writes:

> By the way: what was index version 1?

It was the format used before ccc4feb (Convert the index file
reading/writing to use network byte order., 2005-04-15)

Re: [PATCH 01/14] Extend index to save more flags

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Jakub Narebski <jnareb@gmail.com> writes:
>
>> By the way: what was index version 1?
>
> It was the format used before ccc4feb (Convert the index file
> reading/writing to use network byte order., 2005-04-15)

Sorry, I dug that wrong.  The right one is:

ca9be05 (Make the sha1 of the index file go at the very end of the file., 2005-04-20)

Re: [PATCH 01/14] Extend index to save more flags

[Nguyen Thai Ngoc Duy]

On 9/21/08, Jakub Narebski <jnareb@gmail.com> wrote:
>  > +
>  > +#define CE_EXTENDED_FLAGS (0)
>  > +
>  > +/*
>  > + * Safeguard to avoid saving wrong flags:
>  > + *  - CE_EXTENDED2 won't get saved until its semantic is known
>  > + *  - Bits in 0x0000FFFF have been saved in ce_flags already
>  > + *  - Bits in 0x003F0000 are currently in-memory flags
>  > + */
>  > +#if CE_EXTENDED_FLAGS & 0x80CFFFFF
>  > +#error "CE_EXTENDED_FLAGS out of range"
>  > +#endif
>
>
> I don't quite understand the above fragment (especially with the fact
>  that CE_EXTENDED_FLAGS is defined as (0))...

Because this patch does not introduce any new on-disk flag yet so
CE_EXTENDED_FLAGS remains 0. In the next patch, CE_EXTENDED_FLAGS will
be updated to have CE_NO_CHECKOUT.
-- 
Duy

Re: [PATCH 01/14] Extend index to save more flags

[Jakub Narebski]

On Sun, 21 Sep 2008, Nguyen Thai Ngoc Duy wrote:
> On 9/21/08, Jakub Narebski <jnareb@gmail.com> wrote:
> > > +
> > > +#define CE_EXTENDED_FLAGS (0)
> > > +
> > > +/*
> > > + * Safeguard to avoid saving wrong flags:
> > > + *  - CE_EXTENDED2 won't get saved until its semantic is known
> > > + *  - Bits in 0x0000FFFF have been saved in ce_flags already
> > > + *  - Bits in 0x003F0000 are currently in-memory flags
> > > + */
> > > +#if CE_EXTENDED_FLAGS & 0x80CFFFFF
> > > +#error "CE_EXTENDED_FLAGS out of range"
> > > +#endif
> >
> >
> > I don't quite understand the above fragment (especially with the fact
> >  that CE_EXTENDED_FLAGS is defined as (0))...
> 
> Because this patch does not introduce any new on-disk flag yet so
> CE_EXTENDED_FLAGS remains 0. In the next patch, CE_EXTENDED_FLAGS will
> be updated to have CE_NO_CHECKOUT.

Well, now I understand CE_EXTENDED_FLAGS being (0).

What I still don't understand the pattern it is protected against.  
As I understand it if CE_EXTENDED_FLAGS & 0x0000FFFF it is bad,
because ce_flags saved flags are not extended flags, and 
CE_EXTENDED_FLAGS & 0x003F0000 are in-memory flags.  But why
CE_EXTENDED_FLAGS & 0x80C00000 is bad, and why (if I understand it)
CE_EXTENDED_FLAGS & 0x00300000 is not bad.

-- 
Jakub Narebski
Poland

Re: [PATCH v2 00/14] Sparse checkout

[Santi Béjar]

On Sat, Sep 20, 2008 at 12:01 PM, Nguyễn Thái Ngọc Duy
<pclouds@gmail.com> wrote:
>
> Nguyễn Thái Ngọc Duy (14):
>  Extend index to save more flags
>  Introduce CE_NO_CHECKOUT bit
>  ls-files: add options to support sparse checkout
>  update-index: refactor mark_valid() in preparation for new options
>  update-index: add --checkout/--no-checkout to update CE_NO_CHECKOUT bit
>  ls-files: Add tests for --sparse and friends
>  Prevent diff machinery from examining worktree outside sparse checkout
>  checkout_entry(): CE_NO_CHECKOUT on checked out entries.
>  grep: skip files outside sparse checkout area
>  ls-files: support "sparse patterns", used to form sparse checkout areas
>  unpack_trees(): add support for sparse checkout
>  clone: support sparse checkout with --narrow-path option
>  checkout: add new options to support sparse checkout
>  wt-status: Show orphaned entries in "git status" output
>

I would like to test it, do you have a public repo to fetch it?

Santi

Re: [PATCH v2 00/14] Sparse checkout

[Nguyen Thai Ngoc Duy]

On 9/20/08, Santi Béjar <santi@agolina.net> wrote:
> On Sat, Sep 20, 2008 at 12:01 PM, Nguyễn Thái Ngọc Duy
>  <pclouds@gmail.com> wrote:
>  >
>  > Nguyễn Thái Ngọc Duy (14):
>  >  Extend index to save more flags
>  >  Introduce CE_NO_CHECKOUT bit
>  >  ls-files: add options to support sparse checkout
>  >  update-index: refactor mark_valid() in preparation for new options
>  >  update-index: add --checkout/--no-checkout to update CE_NO_CHECKOUT bit
>  >  ls-files: Add tests for --sparse and friends
>  >  Prevent diff machinery from examining worktree outside sparse checkout
>  >  checkout_entry(): CE_NO_CHECKOUT on checked out entries.
>  >  grep: skip files outside sparse checkout area
>  >  ls-files: support "sparse patterns", used to form sparse checkout areas
>  >  unpack_trees(): add support for sparse checkout
>  >  clone: support sparse checkout with --narrow-path option
>  >  checkout: add new options to support sparse checkout
>  >  wt-status: Show orphaned entries in "git status" output
>  >
>
>
> I would like to test it, do you have a public repo to fetch it?

git://repo.or.cz/git/pclouds.git branch sparse-checkout.
-- 
Duy

Re: [PATCH v2 00/14] Sparse checkout

[Jakub Narebski]

Nguyễn Thái Ngọc Duy <pclouds@gmail.com> writes:

> Main changes from the last round are:
>  - The name is now "sparse checkout"

I guess that 'sparse checkout' would be more recognizable than 'narrow
checkout'.

>  - "git clone --path" => "git clone --narrow-path"
>  - "git checkout --path" => "git checkout --reset-path"

I am not sure about that change, especially the fact that git-clone
and git-checkout use differently named options, because those options
affect clone only as they affect the checkout part of the clone.  One
would think that git-clone = git-init + git-remote add + git-fetch +
git-checkout, and that git-clone would simply pass sparse checkout
flags to git-checkout.

>  - New narrow spec (or "sparse patterns" from now) resembles
>    .gitignore patterns

You mean here that rules for patterns to select which parts of tree
mark as "no-checkout" and/or checkout/leave in checkout are the same
(or nearly the same) as rules for ignoring files, isn't it?

BTW I think that the same rules are used in gitattributes, aren't
they?

>  - "git ls-files" now supports more fine-grained listing. It can now
>    list checkout files, no-checkout files or orphaned (previously
>    "overlay") files. --overlay is gone

Good.  I hope (I haven't read the post yet) that you extended '-t' and
'-v' output (or added similar option for no-checkout bit specially)
for git-ls-files.

>  - "git status" shows orphaned entries and remedies

Good.  I guess (I haven't read the post yet) that git-status displays,
or can be asked to display "orphaned"/"unwanted" files (files which
exists in working repository, but are marked with "no-checkout" bit),
and perhaps simply the fact that we commit/are in narrow checkout
(there is at least one file with "no-checkout" bit set in the index).

> For code changes, significant changes are:

>   [10/14] ls-files: support "sparse patterns", used to form sparse
>           checkout areas

Hmmm...
 
> I hope I have addressed all the issues. If I miss anything, please
> speak up.
 
By the way, pleas do not worry if there would be new round of
comments.  Sparse checkout is a new feature, and I think it is quite
important to get UI (interface) part, like names of configuration
variables and options first, or at least correct them before sparse
checkout hits released version.  It is harder IMVHO to change user
interface (this applies also to non-local repository data) due to
concerns for backwards compatibility; it is I think easier to correct
code.

> Nguyễn Thái Ngọc Duy (14):

Errr... what happened here? For me it doesn't look like correct UTF-8
encoding, but perhaps that it is just my news client (Gnus)...

P.S. Thanks for numbering this series.
-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: [PATCH v2 00/14] Sparse checkout

[Nguyen Thai Ngoc Duy]

On 9/20/08, Jakub Narebski <jnareb@gmail.com> wrote:
>  >  - "git clone --path" => "git clone --narrow-path"
>  >  - "git checkout --path" => "git checkout --reset-path"
>
>
> I am not sure about that change, especially the fact that git-clone
>  and git-checkout use differently named options, because those options
>  affect clone only as they affect the checkout part of the clone.  One
>  would think that git-clone = git-init + git-remote add + git-fetch +
>  git-checkout, and that git-clone would simply pass sparse checkout
>  flags to git-checkout.
>

Johannes sixt said --path was too generic so I changed the name. Hmm..
I did not think the same option name for git-checkout and git-clone
was important, rather worry about people may misunderstand that it is
"narrow clone" (do not fetch objects outside given paths for all
history). Maybe "git clone --narrow-checkout" would be better.
"--reset-path", I think, is a better name though. It would express the
relation compared to --add-path and --remove-path.

>  >  - New narrow spec (or "sparse patterns" from now) resembles
>  >    .gitignore patterns
>
>
> You mean here that rules for patterns to select which parts of tree
>  mark as "no-checkout" and/or checkout/leave in checkout are the same
>  (or nearly the same) as rules for ignoring files, isn't it?

Yes, almost the same, exceptions include "./" support (this may have
worked already for .gitignore, I dunno) and backslash escape for
colons.

>  BTW I think that the same rules are used in gitattributes, aren't
>  they?

They have different implementations. Though the rules may be the same.

>  > Nguyễn Thái NgỠc Duy (14):
>
>  Errr... what happened here? For me it doesn't look like correct UTF-8
>  encoding, but perhaps that it is just my news client (Gnus)...

The cover letter lacks MIME-Version and Content-Type, hmm..
-- 
Duy

Re: [PATCH v2 00/14] Sparse checkout

[Jakub Narebski]

On Sat, 20 Sep 2008, Nguyen Thai Ngoc Duy wrote:
> On 9/20/08, Jakub Narebski <jnareb@gmail.com> wrote:
>>>
>>>  - "git clone --path" => "git clone --narrow-path"
>>>  - "git checkout --path" => "git checkout --reset-path"
>>
>>
>> I am not sure about that change, especially the fact that git-clone
>> and git-checkout use differently named options, because those options
>> affect clone only as they affect the checkout part of the clone.  One
>> would think that git-clone = git-init + git-remote add + git-fetch +
>> git-checkout, and that git-clone would simply pass sparse checkout
>> flags to git-checkout.
>>
> 
> Johannes sixt said --path was too generic so I changed the name. Hmm..
> I did not think the same option name for git-checkout and git-clone
> was important, rather worry about people may misunderstand that it is
> "narrow clone" (do not fetch objects outside given paths for all
> history). Maybe "git clone --narrow-checkout" would be better.

True, I didn't thought that git-clone can have option with explicit
'checkout' in a name, for example --checkout-paths, or --checkout-only,
or --narrow-checkout (although for me this one doesn't look as it
accepts arguments, at least on first glance), or --sparse-checkout
(the same).

> "--reset-path", I think, is a better name though. It would express the
> relation compared to --add-path and --remove-path.

I don't like very much '--reset-path' option name, because it is not
about 'reset' or 'resetting', but about limiting checkout to specified
paths.  Unfortunately --include / --exclude looks like are for ignoring
files, not marking files with "no-checkout" bit etc.

But I am not native English speaker.

Further proposals: --only ("git checkout --only <pattern>") with
--checkout-only as counterpart in git-clone; --limit-to, --sparse.


GNU tar uses --exclude and --exclude-file (with --no-recurse,
--no-wildcards, --no-wildcards-match-slash). wget uses --accept
and --reject for filename patterns, and --include / --exclude for
directories.  Neither looks right for sparse checkout in Git.

>>>  - New narrow spec (or "sparse patterns" from now) resembles
>>>    .gitignore patterns
>>
>>
>> You mean here that rules for patterns to select which parts of tree
>>  mark as "no-checkout" and/or checkout/leave in checkout are the same
>>  (or nearly the same) as rules for ignoring files, isn't it?
> 
> Yes, almost the same, exceptions include "./" support (this may have
> worked already for .gitignore, I dunno) and backslash escape for
> colons.

'./', or rather '/' support works for gitignore: this is the only way
to have pattern which matches only files in given directory, 
nonrecursively.  For example last line in last example on gitignore(5)
man page.

>>  BTW I think that the same rules are used in gitattributes, aren't
>>  they?
> 
> They have different implementations. Though the rules may be the same.

Were you able to reuse either one?

>>> Nguyễn Thái NgỠc Duy (14):
>>
>>  Errr... what happened here? For me it doesn't look like correct UTF-8
>>  encoding, but perhaps that it is just my news client (Gnus)...
> 
> The cover letter lacks MIME-Version and Content-Type, hmm..

Bug in git-format-patch? IIRC --cover-letter was added quite late, and
is quite a new option; some bugs might have been not ironed out yet.

-- 
Jakub Narebski
Poland

Re: Encoding problems with format-patch [Was: [PATCH v2 00/14] Sparse checkout]

[Uwe Kleine-König]

Hello,

On Sat, Sep 20, 2008 at 08:01:27PM +0200, Jakub Narebski wrote:
> On Sat, 20 Sep 2008, Nguyen Thai Ngoc Duy wrote:
> >>> Nguyễn Thái NgỠc Duy (14):
> >>
> >>  Errr... what happened here? For me it doesn't look like correct UTF-8
> >>  encoding, but perhaps that it is just my news client (Gnus)...
> > 
> > The cover letter lacks MIME-Version and Content-Type, hmm..
> 
> Bug in git-format-patch? IIRC --cover-letter was added quite late, and
> is quite a new option; some bugs might have been not ironed out yet.
I hit this kind of problem already, too.

My problem was that format-patch only adds MIME-Version and
Content-Type headers if the commit has non-ascii characters.  If I add a
S-o-b only after the format-patch step I easily forget to add the needed
headers.  IMHO the right fix is to let send-email stop if there is no
encoding related header in the mail but non-ascii characters.

Best regards
Uwe

Re: [PATCH v2 00/14] Sparse checkout

[Nguyen Thai Ngoc Duy]

On 9/21/08, Jakub Narebski <jnareb@gmail.com> wrote:
>  > "--reset-path", I think, is a better name though. It would express the
>  > relation compared to --add-path and --remove-path.
>
>
> I don't like very much '--reset-path' option name, because it is not
>  about 'reset' or 'resetting', but about limiting checkout to specified
>  paths.  Unfortunately --include / --exclude looks like are for ignoring
>  files, not marking files with "no-checkout" bit etc.
>
>  But I am not native English speaker.
>
>  Further proposals: --only ("git checkout --only <pattern>") with
>  --checkout-only as counterpart in git-clone; --limit-to, --sparse.
>
>
>  GNU tar uses --exclude and --exclude-file (with --no-recurse,
>  --no-wildcards, --no-wildcards-match-slash). wget uses --accept
>  and --reject for filename patterns, and --include / --exclude for
>  directories.  Neither looks right for sparse checkout in Git.

I think --exclude and --include are quite good. With Junio's
suggestion "foo-X", how about this?

git clone --sparse-checkout=<patterns> [1]
git checkout --set-sparse=<patterns>
git checkout --include-sparse=<patterns>
git checkout --exclude-sparse=<patterns> [2]

[1] "checkout" is there to avoid being interpreted as "sparse clone"
[2] --narrow-sparse IMO does not tell how the following patterns are
used (is it the set that will be excluded or the target set?),
"exclude" does better.

>  >>  BTW I think that the same rules are used in gitattributes, aren't
>  >>  they?
>  >
>  > They have different implementations. Though the rules may be the same.
>
>
> Were you able to reuse either one?

No. .gitignore is tied to read_directory() while .gitattributes has
attributes attached. So I rolled out another one for index.
-- 
Duy

Re: [PATCH v2 00/14] Sparse checkout

[Junio C Hamano]

"Nguyen Thai Ngoc Duy" <pclouds@gmail.com> writes:

> On 9/21/08, Jakub Narebski <jnareb@gmail.com> wrote:
> ...
>>  >>  BTW I think that the same rules are used in gitattributes, aren't
>>  >>  they?
>>  >
>>  > They have different implementations. Though the rules may be the same.
>>
>> Were you able to reuse either one?
>
> No. .gitignore is tied to read_directory() while .gitattributes has
> attributes attached. So I rolled out another one for index.

I am sorry, but that sounds like a rather lame excuse.  It certainly is
possible to introduce an "ignored" attribute and have .gitattributes file
specify that, instead of having an entry in .gitignore file, if you teach
read_directory() to pay attention to the attributes mechanism.  If we had
from day one that a more generic gitattributes mechanism, I would imagine
we wouldn't even had a separate .gitignore codepath but used the attribute
mechanism throughout the system.

Now I do not think we are ever going to deprecate gitignore and move
everybody to "ignored" attributes, because such a transition would not buy
the end users anything, but it technically is possible and would have been
the right thing to do, if we were building the system from scratch.  We
still could add it as an optional feature (i.e. if a path has the
attribute that says "ignored" or "not ignored", then that determines the
fate of the path, otherwise we look at gitignore).

I wouldn't be surprised if an alternative implementation of your code to
assign "sparseness" to each path internally used "to-be-checked-out"
attribute, and used that attribute to control how ls-files filters its
output.

A better excuse might have been that "I am not reading these patterns from
anywhere but command line", but that got me thinking further.

How would that --narrow-match that is not stored anywhere on the
filesystem but used only for filtering the output be any more useful than
a grep that filters ls-files output in practice?

I would imagine it would be much more useful if .git/info/attributes can
specify "checkout" attribute that is defined like this:

        `checkout`
        ^^^^^^^^^^

        This attribute controls if the path can be left not checked-out to the
        working tree.

        Unset::
                Unsetting the `checkout` marks the path not to be checked out.

        Unspecified::
                A path which does not have any `checkout` attribute specified is
                handled in no special way.

        Any value set to `checkout` is ignored, and git acts as if the
        attribute is left unspecified.

Then whenever a new path enters the index, you _could_ check with the
attribute mechanism to set the CE_NOCHECKOUT flag.  Just like an already
tracked path is not ignored even if it matches .gitignore pattern, a path
without CE_NOCHECKOUT that is in the index is checked out even if it has
checkout attribute Unset.

Hmm?

Re: [PATCH v2 00/14] Sparse checkout

[Nguyen Thai Ngoc Duy]

On 9/21/08, Junio C Hamano <gitster@pobox.com> wrote:
> "Nguyen Thai Ngoc Duy" <pclouds@gmail.com> writes:
>
>  > On 9/21/08, Jakub Narebski <jnareb@gmail.com> wrote:
>
> > ...
>
> >>  >>  BTW I think that the same rules are used in gitattributes, aren't
>  >>  >>  they?
>  >>  >
>  >>  > They have different implementations. Though the rules may be the same.
>  >>
>  >> Were you able to reuse either one?
>  >
>  > No. .gitignore is tied to read_directory() while .gitattributes has
>  > attributes attached. So I rolled out another one for index.
>
>
> I am sorry, but that sounds like a rather lame excuse.  It certainly is
>  possible to introduce an "ignored" attribute and have .gitattributes file
>  specify that, instead of having an entry in .gitignore file, if you teach
>  read_directory() to pay attention to the attributes mechanism.  If we had
>  from day one that a more generic gitattributes mechanism, I would imagine
>  we wouldn't even had a separate .gitignore codepath but used the attribute
>  mechanism throughout the system.
>
>  Now I do not think we are ever going to deprecate gitignore and move
>  everybody to "ignored" attributes, because such a transition would not buy
>  the end users anything, but it technically is possible and would have been
>  the right thing to do, if we were building the system from scratch.  We
>  still could add it as an optional feature (i.e. if a path has the
>  attribute that says "ignored" or "not ignored", then that determines the
>  fate of the path, otherwise we look at gitignore).
>
>  I wouldn't be surprised if an alternative implementation of your code to
>  assign "sparseness" to each path internally used "to-be-checked-out"
>  attribute, and used that attribute to control how ls-files filters its
>  output.
>
>  A better excuse might have been that "I am not reading these patterns from
>  anywhere but command line", but that got me thinking further.

That "from command line" piece makes a bit of difference. For example
patterns separated by colons and backslash escape, but that does not
stop it from reusing attr.c.

>  How would that --narrow-match that is not stored anywhere on the
>  filesystem but used only for filtering the output be any more useful than
>  a grep that filters ls-files output in practice?

Well, it works exactly like 'grep' internally.

>  I would imagine it would be much more useful if .git/info/attributes can
>  specify "checkout" attribute that is defined like this:
>
>         `checkout`
>         ^^^^^^^^^^
>
>         This attribute controls if the path can be left not checked-out to the
>         working tree.
>
>         Unset::
>                 Unsetting the `checkout` marks the path not to be checked out.
>
>         Unspecified::
>                 A path which does not have any `checkout` attribute specified is
>                 handled in no special way.
>
>         Any value set to `checkout` is ignored, and git acts as if the
>         attribute is left unspecified.
>
>  Then whenever a new path enters the index, you _could_ check with the
>  attribute mechanism to set the CE_NOCHECKOUT flag.  Just like an already
>  tracked path is not ignored even if it matches .gitignore pattern, a path
>  without CE_NOCHECKOUT that is in the index is checked out even if it has
>  checkout attribute Unset.
>
>  Hmm?

Well I think people would want to save no-checkout rules eventually.
But I don't know how they want to use it. Will the saved rules be hard
restriction, that no files can be checked out outside defined areas?
Will it be to save a couple of keystrokes,   that is, instead of
typing "--reset-sparse=blah" all the time, now just "--reset-sparse"
and default rules will be applied? Your suggestion would be the third,
applying on new files only.

Anyway I will try to extend attr.c a bit to take input from command
line, then move "sparse patterns" over to use attr.c.
-- 
Duy

Re: [PATCH v2 00/14] Sparse checkout

[Jakub Narebski]

On Sun, 21 Sep 2008, Nguyen Thai Ngoc Duy wrote:
> On 9/21/08, Junio C Hamano <gitster@pobox.com> wrote:

> >  How would that --narrow-match that is not stored anywhere on the
> >  filesystem but used only for filtering the output be any more useful than
> >  a grep that filters ls-files output in practice?
> 
> Well, it works exactly like 'grep' internally.
> 
> >  I would imagine it would be much more useful if .git/info/attributes can
> >  specify "checkout" attribute that is defined like this:
> >
> >         `checkout`
> >         ^^^^^^^^^^
[...]

> >  Then whenever a new path enters the index, you _could_ check with the
> >  attribute mechanism to set the CE_NOCHECKOUT flag.  Just like an already
> >  tracked path is not ignored even if it matches .gitignore pattern, a path
> >  without CE_NOCHECKOUT that is in the index is checked out even if it has
> >  checkout attribute Unset.
> >
> >  Hmm?
> 
> Well I think people would want to save no-checkout rules eventually.
> But I don't know how they want to use it. Will the saved rules be hard
> restriction, that no files can be checked out outside defined areas?
> Will it be to save a couple of keystrokes,   that is, instead of
> typing "--reset-sparse=blah" all the time, now just "--reset-sparse"
> and default rules will be applied? Your suggestion would be the third,
> applying on new files only.
> 
> Anyway I will try to extend attr.c a bit to take input from command
> line, then move "sparse patterns" over to use attr.c.

First, I think that this was Junio asking for discussion more than
for changing the design.

Second, while unifying the "check the match" part of gitignore,
gitattribute and sparse checkout would be IMVHO a good idea, I'm
not sure if trying to use/reuse attr.c literally would be a good
idea, at least not without larger surgery.  AFAIK, IIUC gitattributes
have some limitations, one of which that they are read from working
area (and there is no API for reading from tree); although this could
be enough for `checkout' attribute, which is not that different in
work from `smudge' attribute, or `crlf` attribute.

-- 
Jakub Narebski
Poland

Re: [PATCH v2 00/14] Sparse checkout

[Nguyen Thai Ngoc Duy]

On 9/21/08, Jakub Narebski <jnareb@gmail.com> wrote:
> On Sun, 21 Sep 2008, Nguyen Thai Ngoc Duy wrote:
>  > On 9/21/08, Junio C Hamano <gitster@pobox.com> wrote:
>
>
> > >  How would that --narrow-match that is not stored anywhere on the
>  > >  filesystem but used only for filtering the output be any more useful than
>  > >  a grep that filters ls-files output in practice?
>  >
>  > Well, it works exactly like 'grep' internally.
>  >
>  > >  I would imagine it would be much more useful if .git/info/attributes can
>  > >  specify "checkout" attribute that is defined like this:
>  > >
>  > >         `checkout`
>  > >         ^^^^^^^^^^
>
> [...]
>
>
>  > >  Then whenever a new path enters the index, you _could_ check with the
>  > >  attribute mechanism to set the CE_NOCHECKOUT flag.  Just like an already
>  > >  tracked path is not ignored even if it matches .gitignore pattern, a path
>  > >  without CE_NOCHECKOUT that is in the index is checked out even if it has
>  > >  checkout attribute Unset.
>  > >
>  > >  Hmm?
>  >
>  > Well I think people would want to save no-checkout rules eventually.
>  > But I don't know how they want to use it. Will the saved rules be hard
>  > restriction, that no files can be checked out outside defined areas?
>  > Will it be to save a couple of keystrokes,   that is, instead of
>  > typing "--reset-sparse=blah" all the time, now just "--reset-sparse"
>  > and default rules will be applied? Your suggestion would be the third,
>  > applying on new files only.
>  >
>  > Anyway I will try to extend attr.c a bit to take input from command
>  > line, then move "sparse patterns" over to use attr.c.
>
>
> First, I think that this was Junio asking for discussion more than
>  for changing the design.

I just tried to see if it was feasible. Checking the source again, I
misunderstood  gitattributes/gitingore's leading '/' notion (in a good
way). Leading '/' means './' and that would be fine for
.git{attributes,ignore}. In sparse patterns, leading '/' means
toplevel directory because you may want to checkout some more from a
subdirectory without moving up to toplevel directory. Now
.git{ignore,attributes} and sparse patterns are incompatible, gaah...

>  Second, while unifying the "check the match" part of gitignore,
>  gitattribute and sparse checkout would be IMVHO a good idea, I'm

It is surely good. Optimization like 68492fc (Speedup scanning for
excluded files.) could be applied to .gitattributes too. Now I know
why I was confused when reading the matching part of
.git{attributes,ignore}.
-- 
Duy

Re: [PATCH v2 00/14] Sparse checkout

[Jakub Narebski]

On Sun, 21 Sep 2008, Nguyen Thai Ngoc Duy wrote:
> On 9/21/08, Jakub Narebski <jnareb@gmail.com> wrote:
>> On Sun, 21 Sep 2008, Nguyen Thai Ngoc Duy wrote:
>>> On 9/21/08, Junio C Hamano <gitster@pobox.com> wrote:

> [...] Checking the source again, I
> misunderstood  gitattributes/gitingore's leading '/' notion (in a good
> way). Leading '/' means './' and that would be fine for
> .git{attributes,ignore}. 

By the way it would be nice if gitignore accepted './' as equivalent
to current '/', as this is something I think (from questions here
and on #git) that people expect to work (not reading documentation
carefully enough).  This is something that for example `ls' would use,
or something that `find' returns.

> In sparse patterns, leading '/' means toplevel directory because you
> may want to checkout some more from a subdirectory without moving up
> to toplevel directory. Now .git{ignore,attributes} and sparse patterns
> are incompatible, gaah... 

Well, this doesn't make sense in a _file_, but makes perfect sense when
invoked from _command line_, as option argument.

But I was thinking more about centralizing pattern matching wrt either
full pathname (with prefix stripped, or not), or basename of a file.
If match check is centralized, then if you enhance pattern language (for
selecting which files to mark no-checkout in sparse checkout for example
by allowing '**' which matches also '/' (if you don't go route of 'tar'
with '--wildcards-match-slash' option)), then it would enhance gitignore
patterns and gitattributes patterns too (well, excluding the fact that
they are delimited differently).
 
>>  Second, while unifying the "check the match" part of gitignore,
>>  gitattribute and sparse checkout would be IMVHO a good idea, [...]
> 
> It is surely good. Optimization like 68492fc (Speedup scanning for
> excluded files.) could be applied to .gitattributes too. Now I know
> why I was confused when reading the matching part of
> .git{attributes,ignore}.

And all speedups (well, perhaps not all) would apply to all classes
of matching against patterns as well.
-- 
Jakub Narebski
Poland

Re: [PATCH v2 00/14] Sparse checkout

[Santi Béjar]

On Sun, Sep 21, 2008 at 12:11 PM, Nguyen Thai Ngoc Duy
<pclouds@gmail.com> wrote:
> On 9/21/08, Junio C Hamano <gitster@pobox.com> wrote:
>> "Nguyen Thai Ngoc Duy" <pclouds@gmail.com> writes:
>>
>>  > On 9/21/08, Jakub Narebski <jnareb@gmail.com> wrote:
>>
>> > ...
>>
>> >>  >>  BTW I think that the same rules are used in gitattributes, aren't
>>  >>  >>  they?
>>  >>  >
>>  >>  > They have different implementations. Though the rules may be the same.
>>  >>
>>  >> Were you able to reuse either one?
>>  >
>>  > No. .gitignore is tied to read_directory() while .gitattributes has
>>  > attributes attached. So I rolled out another one for index.
>>
>>
>> I am sorry, but that sounds like a rather lame excuse.  It certainly is
>>  possible to introduce an "ignored" attribute and have .gitattributes file
>>  specify that, instead of having an entry in .gitignore file, if you teach
>>  read_directory() to pay attention to the attributes mechanism.  If we had
>>  from day one that a more generic gitattributes mechanism, I would imagine
>>  we wouldn't even had a separate .gitignore codepath but used the attribute
>>  mechanism throughout the system.
>>
>>  Now I do not think we are ever going to deprecate gitignore and move
>>  everybody to "ignored" attributes, because such a transition would not buy
>>  the end users anything, but it technically is possible and would have been
>>  the right thing to do, if we were building the system from scratch.  We
>>  still could add it as an optional feature (i.e. if a path has the
>>  attribute that says "ignored" or "not ignored", then that determines the
>>  fate of the path, otherwise we look at gitignore).
>>
>>  I wouldn't be surprised if an alternative implementation of your code to
>>  assign "sparseness" to each path internally used "to-be-checked-out"
>>  attribute, and used that attribute to control how ls-files filters its
>>  output.
>>
>>  A better excuse might have been that "I am not reading these patterns from
>>  anywhere but command line", but that got me thinking further.
>
> That "from command line" piece makes a bit of difference. For example
> patterns separated by colons and backslash escape, but that does not
> stop it from reusing attr.c.
>
>>  How would that --narrow-match that is not stored anywhere on the
>>  filesystem but used only for filtering the output be any more useful than
>>  a grep that filters ls-files output in practice?
>
> Well, it works exactly like 'grep' internally.
>
>>  I would imagine it would be much more useful if .git/info/attributes can
>>  specify "checkout" attribute that is defined like this:
>>
>>         `checkout`
>>         ^^^^^^^^^^
>>
>>         This attribute controls if the path can be left not checked-out to the
>>         working tree.
>>
>>         Unset::
>>                 Unsetting the `checkout` marks the path not to be checked out.
>>
>>         Unspecified::
>>                 A path which does not have any `checkout` attribute specified is
>>                 handled in no special way.
>>
>>         Any value set to `checkout` is ignored, and git acts as if the
>>         attribute is left unspecified.
>>
>>  Then whenever a new path enters the index, you _could_ check with the
>>  attribute mechanism to set the CE_NOCHECKOUT flag.  Just like an already
>>  tracked path is not ignored even if it matches .gitignore pattern, a path
>>  without CE_NOCHECKOUT that is in the index is checked out even if it has
>>  checkout attribute Unset.
>>
>>  Hmm?
>
> Well I think people would want to save no-checkout rules eventually.
> But I don't know how they want to use it. Will the saved rules be hard
> restriction, that no files can be checked out outside defined areas?
> Will it be to save a couple of keystrokes,   that is, instead of
> typing "--reset-sparse=blah" all the time, now just "--reset-sparse"
> and default rules will be applied? Your suggestion would be the third,
> applying on new files only.
>
> Anyway I will try to extend attr.c a bit to take input from command
> line, then move "sparse patterns" over to use attr.c.


While I agree that the checkout attr looks like an attribute (so
reusing attr.c is a good idea) and $GIT_DIR/info/gitattributes seems a
good place to specify them, I think it will be better in the config
$GIT_DIR/config. There it is clear that it is a local thing and you
have "git config" to read and write them. Additionally you could have
different patterns in the config (sparse.default, sparse.doc,
sparse.src,...), although maybe it is not very useful.

I think the main UI to sparse checkout should be a default sparse
pattern that is used for "all" commands, like merge, reset, and
checkout. Now it is too easy to escape from the sparse checkout, when
you merge or checkout a branch with new files, when doing a "git reset
--hard" (when you abort a failed merge), or when doing a diff
(specially when you pull).

Santi

Re: [PATCH v2 00/14] Sparse checkout

[Nguyen Thai Ngoc Duy]

On Tue, Sep 23, 2008 at 01:06:47PM +0200, =?ISO-8859-1?Q?Santi_B=E9jar_ wrote:
> While I agree that the checkout attr looks like an attribute (so
> reusing attr.c is a good idea) and $GIT_DIR/info/gitattributes seems a
> good place to specify them, I think it will be better in the config
> $GIT_DIR/config. There it is clear that it is a local thing and you
> have "git config" to read and write them. Additionally you could have
> different patterns in the config (sparse.default, sparse.doc,
> sparse.src,...), although maybe it is not very useful.
> 
> I think the main UI to sparse checkout should be a default sparse
> pattern that is used for "all" commands, like merge, reset, and
> checkout. Now it is too easy to escape from the sparse checkout, when
> you merge or checkout a branch with new files, when doing a "git reset
> --hard" (when you abort a failed merge), or when doing a diff
> (specially when you pull).

It should not escape that easy (except newly added files). There is a
bug in my apply_narrow_spec() that effectively disables sparse
checkkout for other unpack_trees() calls except checkout/clone. Try
the below patch.

diff --git a/unpack-trees.c b/unpack-trees.c
index 10f377c..5bbe016 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -146,8 +146,10 @@ static int apply_narrow_spec(struct unpack_trees_options *o)
 	struct index_state *index = &o->result;
 	int i;
 
+	/*
 	if (!(o->new_narrow_path | o->add_narrow_path | o->remove_narrow_path))
 		return 0;
+	*/
 
 	for (i = 0; i < index->cache_nr; i++) {
 		struct cache_entry *ce = index->cache[i];
-- 
Duy

Re: [PATCH v2 00/14] Sparse checkout

[Nguyen Thai Ngoc Duy]

On 9/23/08, Santi Béjar <santi@agolina.net> wrote:
> While I agree that the checkout attr looks like an attribute (so
>  reusing attr.c is a good idea) and $GIT_DIR/info/gitattributes seems a
>  good place to specify them, I think it will be better in the config
>  $GIT_DIR/config. There it is clear that it is a local thing and you
>  have "git config" to read and write them. Additionally you could have
>  different patterns in the config (sparse.default, sparse.doc,
>  sparse.src,...), although maybe it is not very useful.
>
>  I think the main UI to sparse checkout should be a default sparse
>  pattern that is used for "all" commands, like merge, reset, and
>  checkout. Now it is too easy to escape from the sparse checkout, when
>  you merge or checkout a branch with new files, when doing a "git reset
>  --hard" (when you abort a failed merge), or when doing a diff
>  (specially when you pull).

I have made a patch to save default sparse patterns, something to play
with so we can have better idea how to do it properly.

There is another option --default-sparse in "git clone" and "git
checkout". The option can be used to save default sparse patterns
(specified by --sparse-checkout in "git clone" or --reset-sparse in
"git checkout"). Something like this:

git clone --default-sparse --sparse-checkout=Documentation/ git.git
git checkout --default-sparse --reset-sparse=t/

Default sparse patterns will be used for other unpack_trees()-related
commands like reset, read-tree, merge, pull... For "git checkout" it
will only be used when neither --full, --reset-sparse,
--include-sparse nor --exclude-sparse is present. And it only applies
to newly-added files.

Patch series is in http://repo.or.cz/w/git/pclouds.git (branch
sparse-checkout). Note that it also incorporates fixes and some option
renames.
-- 
Duy

Re: [PATCH v2 00/14] Sparse checkout

[Junio C Hamano]

"Nguyen Thai Ngoc Duy" <pclouds@gmail.com> writes:

> On 9/20/08, Jakub Narebski <jnareb@gmail.com> wrote:
>>  >  - "git clone --path" => "git clone --narrow-path"
>>  >  - "git checkout --path" => "git checkout --reset-path"
>>
>> I am not sure about that change, especially the fact that git-clone
>>  and git-checkout use differently named options, because those options
>>  affect clone only as they affect the checkout part of the clone.  One
>>  would think that git-clone = git-init + git-remote add + git-fetch +
>>  git-checkout, and that git-clone would simply pass sparse checkout
>>  flags to git-checkout.
>>
>
> Johannes sixt said --path was too generic so I changed the name. Hmm..
> I did not think the same option name for git-checkout and git-clone
> was important, rather worry about people may misunderstand that it is
> "narrow clone" (do not fetch objects outside given paths for all
> history). Maybe "git clone --narrow-checkout" would be better.

Be it narrow or sparse, I would agree this round is better than too
generic sounding --path.  Whatever that "limited set of paths that are to
appear in the working tree" is called, it is a good idea to name options
to clarify what effect each option causes on that concept X.  So perhaps:

	--reset-X	resets the checkout set
        --widen-X	widens the checkout set
        --narrow-X	narrows the checkout set

is a good set of options to give "checkout".

You could argue that using "--reset-X" would be more consistent for clone,
but I think the "--reset-" part is redundant, because clone cannot
possibly say "add these" or "subtract those".  I.e. it can only establish
the initial set (which is "--reset-X").  Which leads me to suggest that it
would not be such a good idea to use the same option name for clone and
checkout

With s/X/sparse/, above would give a set of options that does not sound
too bad, I think.  It certainly is better than s/X/narrow/ which would
have a strong "Huh?" factor in "--narrow-narrow" one ;-)

Re: [PATCH v2 00/14] Sparse checkout

[Santi Béjar]

Hi,

  Just some comments.

  When you exclude the .gitignore file all the ignored files are
reported as "Untracked files". I.e, as in:

$git clone $git_url
$ cd git
$ make
$ git checkout --reset-path=Documentation/
$ git status

  When you have local changes it says that it cannot switch branches

$ git checkout --reset-path=Documentation/
error: You have local changes to 'Makefile'; cannot switch branches.

$ git checkout -h
...
    --reset-path <prefixes>
                          reset to new sparse checkout
    --add-path <prefixes>
                          widen checkout area
    --remove-path <prefixes>
                          narrow checkout area

s/prefixes/sparse patterns/

Best regards,

Santi

Re: [PATCH 01/14] Extend index to save more flags

[Jakub Narebski]

On Mon, 22 Sep 2008, Duy Nguyen wrote:
> On 09/22/2008 "Jakub Narebski" <jnareb@gmail.com> wrote:
>>On Sun, 21 Sep 2008, Nguyen Thai Ngoc Duy wrote:
>>> On 9/21/08, Jakub Narebski <jnareb@gmail.com> wrote:
>>>>> +
>>>>> +#define CE_EXTENDED_FLAGS (0)
>>>>> +
>>>>> +/*
>>>>> + * Safeguard to avoid saving wrong flags:
>>>>> + *  - CE_EXTENDED2 won't get saved until its semantic is known
>>>>> + *  - Bits in 0x0000FFFF have been saved in ce_flags already
>>>>> + *  - Bits in 0x003F0000 are currently in-memory flags
>>>>> + */
>>>>> +#if CE_EXTENDED_FLAGS & 0x80CFFFFF
>>>>> +#error "CE_EXTENDED_FLAGS out of range"
>>>>> +#endif
>>>>
>>>>
>>>> I don't quite understand the above fragment (especially with the fact
>>>>  that CE_EXTENDED_FLAGS is defined as (0))...
>>> 
>>> Because this patch does not introduce any new on-disk flag yet so
>>> CE_EXTENDED_FLAGS remains 0. In the next patch, CE_EXTENDED_FLAGS will
>>> be updated to have CE_NO_CHECKOUT.
>> 
>> Well, now I understand CE_EXTENDED_FLAGS being (0).
>> 
>> What I still don't understand the pattern it is protected against.  
>> As I understand it if CE_EXTENDED_FLAGS & 0x0000FFFF it is bad,
>> because ce_flags saved flags are not extended flags, and 
>> CE_EXTENDED_FLAGS & 0x003F0000 are in-memory flags.  But why
>> CE_EXTENDED_FLAGS & 0x80C00000 is bad, and why (if I understand it)
>> CE_EXTENDED_FLAGS & 0x00300000 is not bad.
> 
> Wrong bit computation, should be "#if CE_EXTENDED_FLAGS & 0x803FFFFF".
> Thanks for pointing out. 

So now there is:

Now CE_EXTENDED_FLAGS & 0x803FFFFF is bad because:
  * CE_EXTENDED_FLAGS & 0x0000FFFF are saved flags (not extended)
  * CE_EXTENDED_FLAGS & 0x003F0000 are in-memory flags
  * CE_EXTENDED_FLAGS & 0x80000000 is 'extra flags' bit
    (this is not mentioned in quoted comment; I'm not sure if
     it needs to be or not)

Is that correct?
-- 
Jakub Narebski
Poland

Re: [PATCH 01/14] Extend index to save more flags

[Nguyen Thai Ngoc Duy]

On 9/28/08, Jakub Narebski <jnareb@gmail.com> wrote:
> So now there is:
>
>  Now CE_EXTENDED_FLAGS & 0x803FFFFF is bad because:
>   * CE_EXTENDED_FLAGS & 0x0000FFFF are saved flags (not extended)
>
>   * CE_EXTENDED_FLAGS & 0x003F0000 are in-memory flags
>
>   * CE_EXTENDED_FLAGS & 0x80000000 is 'extra flags' bit
>     (this is not mentioned in quoted comment; I'm not sure if
>      it needs to be or not)
>
>  Is that correct?

Correct.
-- 
Duy