[PATCH/resent 0/9] Sparse checkout (first half)

[PATCH/resent 0/9] Sparse checkout (first half)

[Nguyễn Thái Ngọc Duy]

This is the first half of the series, making git ready for sparse
checkout. The only difference from the last (first half) sent
series is safeguard bitmask fix in 1/9

Nguyễn Thái Ngọc Duy (9):
  Extend index to save more flags
  Introduce CE_NO_CHECKOUT bit
  ls-files: add options to support sparse checkout
  update-index: refactor mark_valid() in preparation for new options
  update-index: add --checkout/--no-checkout to update CE_NO_CHECKOUT
    bit
  ls-files: Add tests for --sparse and friends
  Prevent diff machinery from examining worktree outside sparse
    checkout
  checkout_entry(): CE_NO_CHECKOUT on checked out entries.
  grep: skip files outside sparse checkout area

 .gitignore                            |    1 +
 Documentation/git-checkout.txt        |   34 +++++++++++++++++
 Documentation/git-grep.txt            |    4 +-
 Documentation/git-ls-files.txt        |   24 +++++++++++-
 Documentation/git-update-index.txt    |   13 ++++++
 Makefile                              |    2 +-
 builtin-grep.c                        |    7 +++-
 builtin-ls-files.c                    |   41 ++++++++++++++++++--
 builtin-update-index.c                |   40 ++++++++++++-------
 cache.h                               |   66 +++++++++++++++++++++++++++++++--
 diff-lib.c                            |    5 +-
 diff.c                                |    4 +-
 entry.c                               |    1 +
 read-cache.c                          |   57 ++++++++++++++++++++++------
 t/t2104-update-index-no-checkout.sh   |   36 ++++++++++++++++++
 t/t3004-ls-files-sparse.sh            |   40 ++++++++++++++++++++
 t/t3004/cached.expected               |    5 ++
 t/t3004/deleted.expected              |    1 +
 t/t3004/everything.expected           |   10 +++++
 t/t3004/modified.expected             |    2 +
 t/t3004/no-checkout.expected          |    2 +
 t/t3004/orphaned-no-checkout.expected |    3 +
 t/t3004/orphaned.expected             |    1 +
 t/t3004/others.expected               |    2 +
 t/t3004/sparse-cached.expected        |    3 +
 t/t3004/sparse-everything.expected    |   11 +++++
 test-index-version.c                  |   14 +++++++
 27 files changed, 385 insertions(+), 44 deletions(-)
 create mode 100755 t/t2104-update-index-no-checkout.sh
 create mode 100755 t/t3004-ls-files-sparse.sh
 create mode 100644 t/t3004/cached.expected
 create mode 100644 t/t3004/deleted.expected
 create mode 100644 t/t3004/everything.expected
 create mode 100644 t/t3004/modified.expected
 create mode 100644 t/t3004/no-checkout.expected
 create mode 100644 t/t3004/orphaned-no-checkout.expected
 create mode 100644 t/t3004/orphaned.expected
 create mode 100644 t/t3004/others.expected
 create mode 100644 t/t3004/sparse-cached.expected
 create mode 100644 t/t3004/sparse-everything.expected
 create mode 100644 test-index-version.c

[PATCH 1/9] Extend index to save more flags

[Nguyễn Thái Ngọc Duy]

The on-disk format of index only saves 16 bit flags, nearly all have
been used. The last bit (CE_EXTENDED) is used to for future extension.

This patch extends index entry format to save more flags in future.
The new entry format will be used when CE_EXTENDED bit is 1.

Because older implementation may not understand CE_EXTENDED bit and
misread the new format, if there is any extended entry in index, index
header version will turn 3, which makes it incompatible for older git.
If there is none, header version will return to 2 again.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 cache.h      |   58 ++++++++++++++++++++++++++++++++++++++++++++++++++++++----
 read-cache.c |   51 +++++++++++++++++++++++++++++++++++++++++----------
 2 files changed, 95 insertions(+), 14 deletions(-)

diff --git a/cache.h b/cache.h
index f4b8ddf..5180879 100644
--- a/cache.h
+++ b/cache.h
@@ -109,6 +109,26 @@ struct ondisk_cache_entry {
 	char name[FLEX_ARRAY]; /* more */
 };
 
+/*
+ * This struct is used when CE_EXTENDED bit is 1
+ * The struct must match ondisk_cache_entry exactly from
+ * ctime till flags
+ */
+struct ondisk_cache_entry_extended {
+	struct cache_time ctime;
+	struct cache_time mtime;
+	unsigned int dev;
+	unsigned int ino;
+	unsigned int mode;
+	unsigned int uid;
+	unsigned int gid;
+	unsigned int size;
+	unsigned char sha1[20];
+	unsigned short flags;
+	unsigned short flags2;
+	char name[FLEX_ARRAY]; /* more */
+};
+
 struct cache_entry {
 	unsigned int ce_ctime;
 	unsigned int ce_mtime;
@@ -130,7 +150,15 @@ struct cache_entry {
 #define CE_VALID     (0x8000)
 #define CE_STAGESHIFT 12
 
-/* In-memory only */
+/*
+ * Range 0xFFFF0000 in ce_flags is divided into
+ * two parts: in-memory flags and on-disk ones.
+ * Flags in CE_EXTENDED_FLAGS will get saved on-disk
+ * if you want to save a new flag, add it in
+ * CE_EXTENDED_FLAGS
+ *
+ * In-memory only flags
+ */
 #define CE_UPDATE    (0x10000)
 #define CE_REMOVE    (0x20000)
 #define CE_UPTODATE  (0x40000)
@@ -140,6 +168,24 @@ struct cache_entry {
 #define CE_UNHASHED  (0x200000)
 
 /*
+ * Extended on-disk flags
+ */
+/* CE_EXTENDED2 is for future extension */
+#define CE_EXTENDED2 0x80000000
+
+#define CE_EXTENDED_FLAGS (0)
+
+/*
+ * Safeguard to avoid saving wrong flags:
+ *  - CE_EXTENDED2 won't get saved until its semantic is known
+ *  - Bits in 0x0000FFFF have been saved in ce_flags already
+ *  - Bits in 0x003F0000 are currently in-memory flags
+ */
+#if CE_EXTENDED_FLAGS & 0x803FFFFF
+#error "CE_EXTENDED_FLAGS out of range"
+#endif
+
+/*
  * Copy the sha1 and stat state of a cache entry from one to
  * another. But we never change the name, or the hash state!
  */
@@ -171,7 +217,9 @@ static inline size_t ce_namelen(const struct cache_entry *ce)
 }
 
 #define ce_size(ce) cache_entry_size(ce_namelen(ce))
-#define ondisk_ce_size(ce) ondisk_cache_entry_size(ce_namelen(ce))
+#define ondisk_ce_size(ce) (((ce)->ce_flags & CE_EXTENDED) ? \
+			    ondisk_cache_entry_extended_size(ce_namelen(ce)) : \
+			    ondisk_cache_entry_size(ce_namelen(ce)))
 #define ce_stage(ce) ((CE_STAGEMASK & (ce)->ce_flags) >> CE_STAGESHIFT)
 #define ce_uptodate(ce) ((ce)->ce_flags & CE_UPTODATE)
 #define ce_mark_uptodate(ce) ((ce)->ce_flags |= CE_UPTODATE)
@@ -214,8 +262,10 @@ static inline int ce_to_dtype(const struct cache_entry *ce)
 	(S_ISREG(mode) ? (S_IFREG | ce_permissions(mode)) : \
 	S_ISLNK(mode) ? S_IFLNK : S_ISDIR(mode) ? S_IFDIR : S_IFGITLINK)
 
-#define cache_entry_size(len) ((offsetof(struct cache_entry,name) + (len) + 8) & ~7)
-#define ondisk_cache_entry_size(len) ((offsetof(struct ondisk_cache_entry,name) + (len) + 8) & ~7)
+#define flexible_size(STRUCT,len) ((offsetof(struct STRUCT,name) + (len) + 8) & ~7)
+#define cache_entry_size(len) flexible_size(cache_entry,len)
+#define ondisk_cache_entry_size(len) flexible_size(ondisk_cache_entry,len)
+#define ondisk_cache_entry_extended_size(len) flexible_size(ondisk_cache_entry_extended,len)
 
 struct index_state {
 	struct cache_entry **cache;
diff --git a/read-cache.c b/read-cache.c
index c5a8659..667c36b 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -1096,7 +1096,7 @@ static int verify_hdr(struct cache_header *hdr, unsigned long size)
 
 	if (hdr->hdr_signature != htonl(CACHE_SIGNATURE))
 		return error("bad signature");
-	if (hdr->hdr_version != htonl(2))
+	if (hdr->hdr_version != htonl(2) && hdr->hdr_version != htonl(3))
 		return error("bad index version");
 	SHA1_Init(&c);
 	SHA1_Update(&c, hdr, size - 20);
@@ -1131,6 +1131,7 @@ int read_index(struct index_state *istate)
 static void convert_from_disk(struct ondisk_cache_entry *ondisk, struct cache_entry *ce)
 {
 	size_t len;
+	const char *name;
 
 	ce->ce_ctime = ntohl(ondisk->ctime.sec);
 	ce->ce_mtime = ntohl(ondisk->mtime.sec);
@@ -1143,19 +1144,31 @@ static void convert_from_disk(struct ondisk_cache_entry *ondisk, struct cache_en
 	/* On-disk flags are just 16 bits */
 	ce->ce_flags = ntohs(ondisk->flags);
 
-	/* For future extension: we do not understand this entry yet */
-	if (ce->ce_flags & CE_EXTENDED)
-		die("Unknown index entry format");
 	hashcpy(ce->sha1, ondisk->sha1);
 
 	len = ce->ce_flags & CE_NAMEMASK;
+
+	if (ce->ce_flags & CE_EXTENDED) {
+		struct ondisk_cache_entry_extended *ondisk2;
+		int extended_flags;
+		ondisk2 = (struct ondisk_cache_entry_extended *)ondisk;
+		extended_flags = ntohs(ondisk2->flags2) << 16;
+		/* We do not yet understand any bit out of CE_EXTENDED_FLAGS */
+		if (extended_flags & ~CE_EXTENDED_FLAGS)
+			die("Unknown index entry format %08x", extended_flags);
+		ce->ce_flags |= extended_flags;
+		name = ondisk2->name;
+	}
+	else
+		name = ondisk->name;
+
 	if (len == CE_NAMEMASK)
-		len = strlen(ondisk->name);
+		len = strlen(name);
 	/*
 	 * NEEDSWORK: If the original index is crafted, this copy could
 	 * go unchecked.
 	 */
-	memcpy(ce->name, ondisk->name, len + 1);
+	memcpy(ce->name, name, len + 1);
 }
 
 static inline size_t estimate_cache_size(size_t ondisk_size, unsigned int entries)
@@ -1415,6 +1428,7 @@ static int ce_write_entry(SHA_CTX *c, int fd, struct cache_entry *ce)
 {
 	int size = ondisk_ce_size(ce);
 	struct ondisk_cache_entry *ondisk = xcalloc(1, size);
+	char *name;
 
 	ondisk->ctime.sec = htonl(ce->ce_ctime);
 	ondisk->ctime.nsec = 0;
@@ -1428,7 +1442,15 @@ static int ce_write_entry(SHA_CTX *c, int fd, struct cache_entry *ce)
 	ondisk->size = htonl(ce->ce_size);
 	hashcpy(ondisk->sha1, ce->sha1);
 	ondisk->flags = htons(ce->ce_flags);
-	memcpy(ondisk->name, ce->name, ce_namelen(ce));
+	if (ce->ce_flags & CE_EXTENDED) {
+		struct ondisk_cache_entry_extended *ondisk2;
+		ondisk2 = (struct ondisk_cache_entry_extended *)ondisk;
+		ondisk2->flags2 = htons((ce->ce_flags & CE_EXTENDED_FLAGS) >> 16);
+		name = ondisk2->name;
+	}
+	else
+		name = ondisk->name;
+	memcpy(name, ce->name, ce_namelen(ce));
 
 	return ce_write(c, fd, ondisk, size);
 }
@@ -1437,16 +1459,25 @@ int write_index(const struct index_state *istate, int newfd)
 {
 	SHA_CTX c;
 	struct cache_header hdr;
-	int i, err, removed;
+	int i, err, removed, extended;
 	struct cache_entry **cache = istate->cache;
 	int entries = istate->cache_nr;
 
-	for (i = removed = 0; i < entries; i++)
+	for (i = removed = extended = 0; i < entries; i++) {
 		if (cache[i]->ce_flags & CE_REMOVE)
 			removed++;
 
+		/* reduce extended entries if possible */
+		cache[i]->ce_flags &= ~CE_EXTENDED;
+		if (cache[i]->ce_flags & CE_EXTENDED_FLAGS) {
+			extended++;
+			cache[i]->ce_flags |= CE_EXTENDED;
+		}
+	}
+
 	hdr.hdr_signature = htonl(CACHE_SIGNATURE);
-	hdr.hdr_version = htonl(2);
+	/* for extended format, increase version so older git won't try to read it */
+	hdr.hdr_version = htonl(extended ? 3 : 2);
 	hdr.hdr_entries = htonl(entries - removed);
 
 	SHA1_Init(&c);
-- 
1.6.0.2.488.gf604a

[PATCH 2/9] Introduce CE_NO_CHECKOUT bit

[Nguyễn Thái Ngọc Duy]

This bit is the basis of sparse checkout. If this bit is on, the entry
is outside sparse checkout and therefore should be ignored (similar
to CE_VALID)

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-checkout.txt |   33 +++++++++++++++++++++++++++++++++
 cache.h                        |   10 +++++++++-
 read-cache.c                   |    6 +++---
 3 files changed, 45 insertions(+), 4 deletions(-)

diff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt
index 82e154d..4bd9eba 100644
--- a/Documentation/git-checkout.txt
+++ b/Documentation/git-checkout.txt
@@ -171,6 +171,39 @@ the reflog for HEAD where you were, e.g.
 $ git log -g -2 HEAD
 ------------
 
+Sparse checkout
+---------------
+
+Normally when you checkout a branch, your working directory
+will be fully populated. In some situations, you just need to
+work on certain files, no full checkout is needed. Sparse
+checkout is a mode that limits the checkout area according to your
+needs. With sparse checkout, you can work on a single file, a
+collection of files, a subdirectory or a collection of separated
+subdirectories.
+
+Because sparse checkout uses a new index format, it will be
+incompatible with git prior to 1.6.0 regarding worktree operations.
+Operations that only need access to the repository itself, such as
+clone, push, or pull/fetch from another (normal) repository... should
+not be affected by sparse checkout.
+
+In sparse checkout mode, checkout status of every files in your
+working directory will be recorded in index. If a file is marked
+"no-checkout", it means that file is not needed to be present in
+working directory by user or any git command. When a new file is added
+to index, it will be marked "checkout" unless sparse patterns are
+applied.  Unmerged files are always "checkout". When you checkout new
+files using "git checkout <file>" they will be automatically marked
+"checkout". Other commands such as "git apply" can also checkout new
+files if they are needed.
+
+"No-checkout" status is very similar to "assume-unchanged bit"
+(see linkgit:git-update-index[1]). The main difference between them
+is "assume unchanged" bit just ignores corresponding files in working
+directory while sparse checkout goes a bit farther, remove those files
+when it is safe to do so.
+
 
 EXAMPLES
 --------
diff --git a/cache.h b/cache.h
index 5180879..f13ddbc 100644
--- a/cache.h
+++ b/cache.h
@@ -170,10 +170,11 @@ struct cache_entry {
 /*
  * Extended on-disk flags
  */
+#define CE_NO_CHECKOUT 0x40000000
 /* CE_EXTENDED2 is for future extension */
 #define CE_EXTENDED2 0x80000000
 
-#define CE_EXTENDED_FLAGS (0)
+#define CE_EXTENDED_FLAGS (CE_NO_CHECKOUT)
 
 /*
  * Safeguard to avoid saving wrong flags:
@@ -185,6 +186,9 @@ struct cache_entry {
 #error "CE_EXTENDED_FLAGS out of range"
 #endif
 
+/* "Assume unchanged" mask */
+#define CE_VALID_MASK (CE_VALID | CE_NO_CHECKOUT)
+
 /*
  * Copy the sha1 and stat state of a cache entry from one to
  * another. But we never change the name, or the hash state!
@@ -222,6 +226,10 @@ static inline size_t ce_namelen(const struct cache_entry *ce)
 			    ondisk_cache_entry_size(ce_namelen(ce)))
 #define ce_stage(ce) ((CE_STAGEMASK & (ce)->ce_flags) >> CE_STAGESHIFT)
 #define ce_uptodate(ce) ((ce)->ce_flags & CE_UPTODATE)
+#define ce_no_checkout(ce) ((ce)->ce_flags & CE_NO_CHECKOUT)
+#define ce_checkout(ce) (!ce_no_checkout(ce))
+#define ce_mark_no_checkout(ce) ((ce)->ce_flags |= CE_NO_CHECKOUT)
+#define ce_mark_checkout(ce) ((ce)->ce_flags &= ~CE_NO_CHECKOUT)
 #define ce_mark_uptodate(ce) ((ce)->ce_flags |= CE_UPTODATE)
 
 #define ce_permissions(mode) (((mode) & 0100) ? 0755 : 0644)
diff --git a/read-cache.c b/read-cache.c
index 667c36b..e965a4c 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -254,7 +254,7 @@ int ie_match_stat(const struct index_state *istate,
 	 * If it's marked as always valid in the index, it's
 	 * valid whatever the checked-out copy says.
 	 */
-	if (!ignore_valid && (ce->ce_flags & CE_VALID))
+	if (!ignore_valid && (ce->ce_flags & CE_VALID_MASK))
 		return 0;
 
 	changed = ce_match_stat_basic(ce, st);
@@ -962,10 +962,10 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,
 		return ce;
 
 	/*
-	 * CE_VALID means the user promised us that the change to
+	 * CE_VALID_MASK means the user promised us that the change to
 	 * the work tree does not matter and told us not to worry.
 	 */
-	if (!ignore_valid && (ce->ce_flags & CE_VALID)) {
+	if (!ignore_valid && (ce->ce_flags & CE_VALID_MASK)) {
 		ce_mark_uptodate(ce);
 		return ce;
 	}
-- 
1.6.0.2.488.gf604a

[PATCH 3/9] ls-files: add options to support sparse checkout

[Nguyễn Thái Ngọc Duy]

The first option to be introduced is --sparse, which puts ls-files
in "sparse mode". In this mode, cached entries are divided into
 - checkout entries: shown by --cached (new behavior with --sparse)
 - no-checkout entries: show by --no-checkout (new option)
 - orphaned entries: shown by --orphaned (new option)

Orphaned entries are themselves no-checkout ones but for some reasons
still be present in working directory.

While at it, fix "--deleted" running out of checkout area.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-ls-files.txt |   24 +++++++++++++++++++++-
 builtin-ls-files.c             |   41 ++++++++++++++++++++++++++++++++++++---
 2 files changed, 59 insertions(+), 6 deletions(-)

diff --git a/Documentation/git-ls-files.txt b/Documentation/git-ls-files.txt
index 9f85d60..1de68e2 100644
--- a/Documentation/git-ls-files.txt
+++ b/Documentation/git-ls-files.txt
@@ -10,8 +10,9 @@ SYNOPSIS
 --------
 [verse]
 'git ls-files' [-z] [-t] [-v]
-		(--[cached|deleted|others|ignored|stage|unmerged|killed|modified])\*
+		(--[cached|deleted|others|ignored|stage|unmerged|killed|modified|orphaned|no-checkout])\*
 		(-[c|d|o|i|s|u|k|m])\*
+		[--sparse]
 		[-x <pattern>|--exclude=<pattern>]
 		[-X <file>|--exclude-from=<file>]
 		[--exclude-per-directory=<file>]
@@ -32,7 +33,9 @@ OPTIONS
 -------
 -c::
 --cached::
-	Show cached files in the output (default)
+	Show cached files in the output (default). When used with --sparse,
+	show only cached files that are marked "checkout", no-checkout
+	entries will be excluded.
 
 -d::
 --deleted::
@@ -72,6 +75,21 @@ OPTIONS
 	to file/directory conflicts for checkout-index to
 	succeed.
 
+--no-checkout::
+	Show no-checkout entries. This option implies --sparse.
+
+--orphaned::
+	Show orphaned entries. Orphaned entries are no-checkout
+	entries that are present in working directory. This option
+	implies --sparse.
+
+--sparse::
+	When --sparse is passed, cached files will be divided into two
+	parts: checkout entries and no-checkout entries.
+	--cached will only show checkout entries.
+	No-checkout entries can be shown using --orphaned or
+	--no-checkout (or both).
+
 -z::
 	\0 line termination on output.
 
@@ -107,6 +125,8 @@ OPTIONS
 	Identify the file status with the following tags (followed by
 	a space) at the start of each line:
 	H::	cached
+	-::	no-checkout entries
+	O::	orphaned entries
 	M::	unmerged
 	R::	removed/deleted
 	C::	modified/changed
diff --git a/builtin-ls-files.c b/builtin-ls-files.c
index 068f424..873de15 100644
--- a/builtin-ls-files.c
+++ b/builtin-ls-files.c
@@ -20,6 +20,9 @@ static int show_unmerged;
 static int show_modified;
 static int show_killed;
 static int show_valid_bit;
+static int show_orphaned;
+static int show_no_checkout;
+static int sparse_checkout;
 static int line_terminator = '\n';
 
 static int prefix_len;
@@ -35,6 +38,8 @@ static const char *tag_removed = "";
 static const char *tag_other = "";
 static const char *tag_killed = "";
 static const char *tag_modified = "";
+static const char *tag_orphaned = "";
+static const char *tag_no_checkout = "";
 
 
 /*
@@ -235,7 +240,7 @@ static void show_files(struct dir_struct *dir, const char *prefix)
 		if (show_killed)
 			show_killed_files(dir);
 	}
-	if (show_cached | show_stage) {
+	if (show_cached | show_stage | show_orphaned | show_no_checkout) {
 		for (i = 0; i < active_nr; i++) {
 			struct cache_entry *ce = active_cache[i];
 			int dtype = ce_to_dtype(ce);
@@ -245,6 +250,16 @@ static void show_files(struct dir_struct *dir, const char *prefix)
 				continue;
 			if (ce->ce_flags & CE_UPDATE)
 				continue;
+			if (sparse_checkout && ce_no_checkout(ce)) {
+				struct stat st;
+				if (show_no_checkout)
+					show_ce_entry(tag_no_checkout, ce);
+				if (show_orphaned && !lstat(ce->name, &st))
+					show_ce_entry(tag_orphaned, ce);
+				continue;
+			}
+			if (!(show_cached | show_stage))
+				continue;
 			show_ce_entry(ce_stage(ce) ? tag_unmerged : tag_cached, ce);
 		}
 	}
@@ -257,7 +272,7 @@ static void show_files(struct dir_struct *dir, const char *prefix)
 			if (excluded(dir, ce->name, &dtype) != dir->show_ignored)
 				continue;
 			err = lstat(ce->name, &st);
-			if (show_deleted && err)
+			if (show_deleted && err && ce_checkout(ce))
 				show_ce_entry(tag_removed, ce);
 			if (show_modified && ce_modified(ce, &st, 0))
 				show_ce_entry(tag_modified, ce);
@@ -423,7 +438,8 @@ int report_path_error(const char *ps_matched, const char **pathspec, int prefix_
 }
 
 static const char ls_files_usage[] =
-	"git ls-files [-z] [-t] [-v] (--[cached|deleted|others|stage|unmerged|killed|modified])* "
+	"git ls-files [-z] [-t] [-v] (--[cached|deleted|others|stage|unmerged|killed|modified|orphaned|no-checkout])* "
+	"[ --sparse ] "
 	"[ --ignored ] [--exclude=<pattern>] [--exclude-from=<file>] "
 	"[ --exclude-per-directory=<filename> ] [--exclude-standard] "
 	"[--full-name] [--abbrev] [--] [<file>]*";
@@ -457,6 +473,8 @@ int cmd_ls_files(int argc, const char **argv, const char *prefix)
 			tag_modified = "C ";
 			tag_other = "? ";
 			tag_killed = "K ";
+			tag_orphaned = "O ";
+			tag_no_checkout = "- ";
 			if (arg[1] == 'v')
 				show_valid_bit = 1;
 			continue;
@@ -465,6 +483,21 @@ int cmd_ls_files(int argc, const char **argv, const char *prefix)
 			show_cached = 1;
 			continue;
 		}
+		if (!strcmp(arg, "--sparse")) {
+			sparse_checkout = 1;
+			continue;
+		}
+		if (!strcmp(arg, "--orphaned")) {
+			show_orphaned = 1;
+			sparse_checkout = 1;
+			require_work_tree = 1;
+			continue;
+		}
+		if (!strcmp(arg, "--no-checkout")) {
+			show_no_checkout = 1;
+			sparse_checkout = 1;
+			continue;
+		}
 		if (!strcmp(arg, "-d") || !strcmp(arg, "--deleted")) {
 			show_deleted = 1;
 			continue;
@@ -593,7 +626,7 @@ int cmd_ls_files(int argc, const char **argv, const char *prefix)
 
 	/* With no flags, we default to showing the cached files */
 	if (!(show_stage | show_deleted | show_others | show_unmerged |
-	      show_killed | show_modified))
+	      show_killed | show_modified | show_orphaned | show_no_checkout))
 		show_cached = 1;
 
 	read_cache();
-- 
1.6.0.2.488.gf604a

[PATCH 4/9] update-index: refactor mark_valid() in preparation for new options

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin-update-index.c |   24 ++++++++++--------------
 1 files changed, 10 insertions(+), 14 deletions(-)

diff --git a/builtin-update-index.c b/builtin-update-index.c
index 417f972..ae94739 100644
--- a/builtin-update-index.c
+++ b/builtin-update-index.c
@@ -24,8 +24,8 @@ static int info_only;
 static int force_remove;
 static int verbose;
 static int mark_valid_only;
-#define MARK_VALID 1
-#define UNMARK_VALID 2
+#define MARK_FLAG 1
+#define UNMARK_FLAG 2
 
 static void report(const char *fmt, ...)
 {
@@ -40,19 +40,15 @@ static void report(const char *fmt, ...)
 	va_end(vp);
 }
 
-static int mark_valid(const char *path)
+static int mark_ce_flags(const char *path, int flag, int mark)
 {
 	int namelen = strlen(path);
 	int pos = cache_name_pos(path, namelen);
 	if (0 <= pos) {
-		switch (mark_valid_only) {
-		case MARK_VALID:
-			active_cache[pos]->ce_flags |= CE_VALID;
-			break;
-		case UNMARK_VALID:
-			active_cache[pos]->ce_flags &= ~CE_VALID;
-			break;
-		}
+		if (mark)
+			active_cache[pos]->ce_flags |= flag;
+		else
+			active_cache[pos]->ce_flags &= ~flag;
 		cache_tree_invalidate_path(active_cache_tree, path);
 		active_cache_changed = 1;
 		return 0;
@@ -276,7 +272,7 @@ static void update_one(const char *path, const char *prefix, int prefix_length)
 		goto free_return;
 	}
 	if (mark_valid_only) {
-		if (mark_valid(p))
+		if (mark_ce_flags(p, CE_VALID, mark_valid_only == MARK_FLAG))
 			die("Unable to mark file %s", path);
 		goto free_return;
 	}
@@ -649,11 +645,11 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)
 				continue;
 			}
 			if (!strcmp(path, "--assume-unchanged")) {
-				mark_valid_only = MARK_VALID;
+				mark_valid_only = MARK_FLAG;
 				continue;
 			}
 			if (!strcmp(path, "--no-assume-unchanged")) {
-				mark_valid_only = UNMARK_VALID;
+				mark_valid_only = UNMARK_FLAG;
 				continue;
 			}
 			if (!strcmp(path, "--info-only")) {
-- 
1.6.0.2.488.gf604a

[PATCH 5/9] update-index: add --checkout/--no-checkout to update CE_NO_CHECKOUT bit

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 .gitignore                          |    1 +
 Documentation/git-checkout.txt      |    3 +-
 Documentation/git-update-index.txt  |   13 ++++++++++++
 Makefile                            |    2 +-
 builtin-update-index.c              |   16 ++++++++++++++-
 t/t2104-update-index-no-checkout.sh |   36 +++++++++++++++++++++++++++++++++++
 test-index-version.c                |   14 +++++++++++++
 7 files changed, 82 insertions(+), 3 deletions(-)
 create mode 100755 t/t2104-update-index-no-checkout.sh
 create mode 100644 test-index-version.c

diff --git a/.gitignore b/.gitignore
index bbaf9de..0c35577 100644
--- a/.gitignore
+++ b/.gitignore
@@ -147,6 +147,7 @@ test-date
 test-delta
 test-dump-cache-tree
 test-genrandom
+test-index-version
 test-match-trees
 test-parse-options
 test-path-utils
diff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt
index 4bd9eba..2b344e1 100644
--- a/Documentation/git-checkout.txt
+++ b/Documentation/git-checkout.txt
@@ -196,7 +196,8 @@ to index, it will be marked "checkout" unless sparse patterns are
 applied.  Unmerged files are always "checkout". When you checkout new
 files using "git checkout <file>" they will be automatically marked
 "checkout". Other commands such as "git apply" can also checkout new
-files if they are needed.
+files if they are needed. linkgit:git-update-index[1] can be used to
+update "checkout/no-checkout" status in index.
 
 "No-checkout" status is very similar to "assume-unchanged bit"
 (see linkgit:git-update-index[1]). The main difference between them
diff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt
index 1d9d81a..ec03e05 100644
--- a/Documentation/git-update-index.txt
+++ b/Documentation/git-update-index.txt
@@ -15,6 +15,7 @@ SYNOPSIS
 	     [--cacheinfo <mode> <object> <file>]\*
 	     [--chmod=(+|-)x]
 	     [--assume-unchanged | --no-assume-unchanged]
+	     [--checkout | --no-checkout]
 	     [--ignore-submodules]
 	     [--really-refresh] [--unresolve] [--again | -g]
 	     [--info-only] [--index-info]
@@ -99,6 +100,18 @@ in the index e.g. when merging in a commit;
 thus, in case the assumed-untracked file is changed upstream,
 you will need to handle the situation manually.
 
+--checkout::
+--no-checkout::
+	When one of these flags is specified, the object name recorded
+	for the paths are not updated. Instead, these options
+	set and unset the "no-checkout" bit for the paths. This
+	bit is used for marking files for sparse checkout. If
+	a path is marked "no-checkout", then it should not be
+	checked out unless requested by user or needed for a git
+	command to function.
+	See linkgit:git-checkout[1] for more information about
+	sparse checkout.
+
 -g::
 --again::
 	Runs 'git-update-index' itself on the paths whose index
diff --git a/Makefile b/Makefile
index e0c03c3..edb33cb 100644
--- a/Makefile
+++ b/Makefile
@@ -1327,7 +1327,7 @@ endif
 
 ### Testing rules
 
-TEST_PROGRAMS = test-chmtime$X test-genrandom$X test-date$X test-delta$X test-sha1$X test-match-trees$X test-parse-options$X test-path-utils$X
+TEST_PROGRAMS = test-chmtime$X test-genrandom$X test-date$X test-delta$X test-sha1$X test-match-trees$X test-parse-options$X test-path-utils$X test-index-version$X
 
 all:: $(TEST_PROGRAMS)
 
diff --git a/builtin-update-index.c b/builtin-update-index.c
index ae94739..7514aff 100644
--- a/builtin-update-index.c
+++ b/builtin-update-index.c
@@ -24,6 +24,7 @@ static int info_only;
 static int force_remove;
 static int verbose;
 static int mark_valid_only;
+static int mark_no_checkout_only;
 #define MARK_FLAG 1
 #define UNMARK_FLAG 2
 
@@ -276,6 +277,11 @@ static void update_one(const char *path, const char *prefix, int prefix_length)
 			die("Unable to mark file %s", path);
 		goto free_return;
 	}
+	if (mark_no_checkout_only) {
+		if (mark_ce_flags(p, CE_NO_CHECKOUT, mark_no_checkout_only == MARK_FLAG))
+			die("Unable to mark file %s", path);
+		goto free_return;
+	}
 
 	if (force_remove) {
 		if (remove_file_from_cache(p))
@@ -386,7 +392,7 @@ static void read_index_info(int line_termination)
 }
 
 static const char update_index_usage[] =
-"git update-index [-q] [--add] [--replace] [--remove] [--unmerged] [--refresh] [--really-refresh] [--cacheinfo] [--chmod=(+|-)x] [--assume-unchanged] [--info-only] [--force-remove] [--stdin] [--index-info] [--unresolve] [--again | -g] [--ignore-missing] [-z] [--verbose] [--] <file>...";
+"git update-index [-q] [--add] [--replace] [--remove] [--unmerged] [--refresh] [--really-refresh] [--cacheinfo] [--chmod=(+|-)x] [--assume-unchanged] [--checkout|--no-checkout] [--info-only] [--force-remove] [--stdin] [--index-info] [--unresolve] [--again | -g] [--ignore-missing] [-z] [--verbose] [--] <file>...";
 
 static unsigned char head_sha1[20];
 static unsigned char merge_head_sha1[20];
@@ -652,6 +658,14 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)
 				mark_valid_only = UNMARK_FLAG;
 				continue;
 			}
+			if (!strcmp(path, "--checkout")) {
+				mark_no_checkout_only = UNMARK_FLAG;
+				continue;
+			}
+			if (!strcmp(path, "--no-checkout")) {
+				mark_no_checkout_only = MARK_FLAG;
+				continue;
+			}
 			if (!strcmp(path, "--info-only")) {
 				info_only = 1;
 				continue;
diff --git a/t/t2104-update-index-no-checkout.sh b/t/t2104-update-index-no-checkout.sh
new file mode 100755
index 0000000..be9f913
--- /dev/null
+++ b/t/t2104-update-index-no-checkout.sh
@@ -0,0 +1,36 @@
+#!/bin/sh
+#
+# Copyright (c) 2008 Nguyễn Thái Ngọc Duy
+#
+
+test_description='git update-index no-checkout bits (a.k.a sparse checkout)'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+	mkdir sub &&
+	touch 1 2 sub/1 sub/2 &&
+	git add 1 2 sub/1 sub/2
+'
+
+test_expect_success 'index is at version 2' '
+	test "$(test-index-version < .git/index)" = 2
+'
+
+test_expect_success 'update-index --no-checkout' '
+	git update-index --no-checkout 1 sub/1 &&
+	test -z "$(git ls-files --sparse|grep 1)"'
+
+test_expect_success 'index is at version 3 after having some no-checkout entries' '
+	test "$(test-index-version < .git/index)" = 3
+'
+
+test_expect_success 'update-index --checkout' '
+	git update-index --checkout 1 sub/1 &&
+	test "$(git ls-files)" = "$(git ls-files --sparse)"'
+
+test_expect_success 'index version is back to 2 when there is no no-checkout entry' '
+	test "$(test-index-version < .git/index)" = 2
+'
+
+test_done
diff --git a/test-index-version.c b/test-index-version.c
new file mode 100644
index 0000000..bfaad9e
--- /dev/null
+++ b/test-index-version.c
@@ -0,0 +1,14 @@
+#include "cache.h"
+
+int main(int argc, const char **argv)
+{
+	struct cache_header hdr;
+	int version;
+
+	memset(&hdr,0,sizeof(hdr));
+	if (read(0, &hdr, sizeof(hdr)) != sizeof(hdr))
+		return 0;
+	version = ntohl(hdr.hdr_version);
+	printf("%d\n", version);
+	return 0;
+}
-- 
1.6.0.2.488.gf604a

[PATCH 6/9] ls-files: Add tests for --sparse and friends

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 t/t3004-ls-files-sparse.sh            |   40 +++++++++++++++++++++++++++++++++
 t/t3004/cached.expected               |    5 ++++
 t/t3004/deleted.expected              |    1 +
 t/t3004/everything.expected           |   10 ++++++++
 t/t3004/modified.expected             |    2 +
 t/t3004/no-checkout.expected          |    2 +
 t/t3004/orphaned-no-checkout.expected |    3 ++
 t/t3004/orphaned.expected             |    1 +
 t/t3004/others.expected               |    2 +
 t/t3004/sparse-cached.expected        |    3 ++
 t/t3004/sparse-everything.expected    |   11 +++++++++
 11 files changed, 80 insertions(+), 0 deletions(-)
 create mode 100755 t/t3004-ls-files-sparse.sh
 create mode 100644 t/t3004/cached.expected
 create mode 100644 t/t3004/deleted.expected
 create mode 100644 t/t3004/everything.expected
 create mode 100644 t/t3004/modified.expected
 create mode 100644 t/t3004/no-checkout.expected
 create mode 100644 t/t3004/orphaned-no-checkout.expected
 create mode 100644 t/t3004/orphaned.expected
 create mode 100644 t/t3004/others.expected
 create mode 100644 t/t3004/sparse-cached.expected
 create mode 100644 t/t3004/sparse-everything.expected

diff --git a/t/t3004-ls-files-sparse.sh b/t/t3004-ls-files-sparse.sh
new file mode 100755
index 0000000..ec2c869
--- /dev/null
+++ b/t/t3004-ls-files-sparse.sh
@@ -0,0 +1,40 @@
+#!/bin/sh
+
+test_description="test ls-files in --sparse mode"
+
+. test-lib.sh
+
+test_ls_files() {
+	T=../t3004/$1.expected
+	shift
+	test_expect_success "ls-files $*" "git ls-files $* > result && test_cmp $T result"
+}
+
+test_expect_success 'setup' '
+	touch other orphaned no-checkout cached modified deleted &&
+	git add orphaned no-checkout cached modified deleted &&
+	git update-index --no-checkout orphaned no-checkout &&
+	echo modified >> modified &&
+	rm no-checkout deleted
+'
+
+test_ls_files cached
+test_ls_files cached --cached
+test_ls_files sparse-cached --sparse
+test_ls_files sparse-cached --sparse --cached
+test_ls_files no-checkout --no-checkout
+test_ls_files no-checkout --sparse --no-checkout
+test_ls_files orphaned --orphaned
+test_ls_files orphaned --sparse --orphaned
+test_ls_files orphaned-no-checkout -v --no-checkout --orphaned
+test_ls_files orphaned-no-checkout -v --sparse --no-checkout --orphaned
+test_ls_files deleted --deleted
+test_ls_files deleted --sparse --deleted
+test_ls_files modified --modified
+test_ls_files modified --sparse --modified
+test_ls_files others --others
+test_ls_files others --sparse --others
+test_ls_files everything -v --cached --deleted --modified --others
+test_ls_files sparse-everything -v --cached --no-checkout --orphaned --deleted --modified --others
+
+test_done
\ No newline at end of file
diff --git a/t/t3004/cached.expected b/t/t3004/cached.expected
new file mode 100644
index 0000000..6fd0c78
--- /dev/null
+++ b/t/t3004/cached.expected
@@ -0,0 +1,5 @@
+cached
+deleted
+modified
+no-checkout
+orphaned
diff --git a/t/t3004/deleted.expected b/t/t3004/deleted.expected
new file mode 100644
index 0000000..71779d2
--- /dev/null
+++ b/t/t3004/deleted.expected
@@ -0,0 +1 @@
+deleted
diff --git a/t/t3004/everything.expected b/t/t3004/everything.expected
new file mode 100644
index 0000000..6000328
--- /dev/null
+++ b/t/t3004/everything.expected
@@ -0,0 +1,10 @@
+? other
+? result
+H cached
+H deleted
+H modified
+H no-checkout
+H orphaned
+R deleted
+C deleted
+C modified
diff --git a/t/t3004/modified.expected b/t/t3004/modified.expected
new file mode 100644
index 0000000..644a96e
--- /dev/null
+++ b/t/t3004/modified.expected
@@ -0,0 +1,2 @@
+deleted
+modified
diff --git a/t/t3004/no-checkout.expected b/t/t3004/no-checkout.expected
new file mode 100644
index 0000000..b2a429f
--- /dev/null
+++ b/t/t3004/no-checkout.expected
@@ -0,0 +1,2 @@
+no-checkout
+orphaned
diff --git a/t/t3004/orphaned-no-checkout.expected b/t/t3004/orphaned-no-checkout.expected
new file mode 100644
index 0000000..d687ef0
--- /dev/null
+++ b/t/t3004/orphaned-no-checkout.expected
@@ -0,0 +1,3 @@
+- no-checkout
+- orphaned
+O orphaned
diff --git a/t/t3004/orphaned.expected b/t/t3004/orphaned.expected
new file mode 100644
index 0000000..571b267
--- /dev/null
+++ b/t/t3004/orphaned.expected
@@ -0,0 +1 @@
+orphaned
diff --git a/t/t3004/others.expected b/t/t3004/others.expected
new file mode 100644
index 0000000..bf5bf2b
--- /dev/null
+++ b/t/t3004/others.expected
@@ -0,0 +1,2 @@
+other
+result
diff --git a/t/t3004/sparse-cached.expected b/t/t3004/sparse-cached.expected
new file mode 100644
index 0000000..3453483
--- /dev/null
+++ b/t/t3004/sparse-cached.expected
@@ -0,0 +1,3 @@
+cached
+deleted
+modified
diff --git a/t/t3004/sparse-everything.expected b/t/t3004/sparse-everything.expected
new file mode 100644
index 0000000..5df0599
--- /dev/null
+++ b/t/t3004/sparse-everything.expected
@@ -0,0 +1,11 @@
+? other
+? result
+H cached
+H deleted
+H modified
+- no-checkout
+- orphaned
+O orphaned
+R deleted
+C deleted
+C modified
-- 
1.6.0.2.488.gf604a

[PATCH 7/9] Prevent diff machinery from examining worktree outside sparse checkout

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 diff-lib.c |    5 +++--
 diff.c     |    4 +++-
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/diff-lib.c b/diff-lib.c
index ae96c64..992280b 100644
--- a/diff-lib.c
+++ b/diff-lib.c
@@ -161,7 +161,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)
 				continue;
 		}
 
-		if (ce_uptodate(ce))
+		if (ce_uptodate(ce) || ce_no_checkout(ce))
 			continue;
 
 		changed = check_removed(ce, &st);
@@ -348,6 +348,8 @@ static void do_oneway_diff(struct unpack_trees_options *o,
 	struct rev_info *revs = cbdata->revs;
 	int match_missing, cached;
 
+	/* if the entry is not checked out, don't examine work tree */
+	cached = o->index_only || (idx && ce_no_checkout(idx));
 	/*
 	 * Backward compatibility wart - "diff-index -m" does
 	 * not mean "do not ignore merges", but "match_missing".
@@ -355,7 +357,6 @@ static void do_oneway_diff(struct unpack_trees_options *o,
 	 * But with the revision flag parsing, that's found in
 	 * "!revs->ignore_merges".
 	 */
-	cached = o->index_only;
 	match_missing = !revs->ignore_merges;
 
 	if (cached && idx && ce_stage(idx)) {
diff --git a/diff.c b/diff.c
index b001d7b..c001e2c 100644
--- a/diff.c
+++ b/diff.c
@@ -1798,8 +1798,10 @@ static int reuse_worktree_file(const char *name, const unsigned char *sha1, int
 
 	/*
 	 * If ce matches the file in the work tree, we can reuse it.
+	 * For sparse checkout case, ce_uptodate() may be true although
+	 * the file may or may not exist in the work tree.
 	 */
-	if (ce_uptodate(ce) ||
+	if ((ce_uptodate(ce) && ce_checkout(ce)) ||
 	    (!lstat(name, &st) && !ce_match_stat(ce, &st, 0)))
 		return 1;
 
-- 
1.6.0.2.488.gf604a

[PATCH 8/9] checkout_entry(): CE_NO_CHECKOUT on checked out entries.

[Nguyễn Thái Ngọc Duy]

With this you can just do "git checkout some-files" to
widen your checkout. One caveat though: caller must save
the index.

For all of its callers (unpack_trees(), checkout-index, checkout
and apply), only "git apply" does not write index back.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 entry.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/entry.c b/entry.c
index aa2ee46..305f8d3 100644
--- a/entry.c
+++ b/entry.c
@@ -230,5 +230,6 @@ int checkout_entry(struct cache_entry *ce, const struct checkout *state, char *t
 	} else if (state->not_new)
 		return 0;
 	create_directories(path, state);
+	ce_mark_checkout(ce);
 	return write_entry(ce, path, state, 0);
 }
-- 
1.6.0.2.488.gf604a

[PATCH 9/9] grep: skip files outside sparse checkout area

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-grep.txt |    4 +++-
 builtin-grep.c             |    7 ++++++-
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt
index fa4d133..ee359c9 100644
--- a/Documentation/git-grep.txt
+++ b/Documentation/git-grep.txt
@@ -24,7 +24,9 @@ SYNOPSIS
 DESCRIPTION
 -----------
 Look for specified patterns in the working tree files, blobs
-registered in the index file, or given tree objects.
+registered in the index file, or given tree objects. By default
+it will search in the working tree files. When in sparse checkout
+mode, it only searches checked-out files.
 
 
 OPTIONS
diff --git a/builtin-grep.c b/builtin-grep.c
index 3a51662..d5507d7 100644
--- a/builtin-grep.c
+++ b/builtin-grep.c
@@ -343,6 +343,8 @@ static int external_grep(struct grep_opt *opt, const char **paths, int cached)
 			continue;
 		if (!pathspec_matches(paths, ce->name))
 			continue;
+		if (ce_no_checkout(ce))
+			continue;
 		name = ce->name;
 		if (name[0] == '-') {
 			int len = ce_namelen(ce);
@@ -404,8 +406,11 @@ static int grep_cache(struct grep_opt *opt, const char **paths, int cached)
 				continue;
 			hit |= grep_sha1(opt, ce->sha1, ce->name, 0);
 		}
-		else
+		else {
+			if (ce_no_checkout(ce))
+				continue;
 			hit |= grep_file(opt, ce->name);
+		}
 		if (ce_stage(ce)) {
 			do {
 				nr++;
-- 
1.6.0.2.488.gf604a

Re: [PATCH 5/9] update-index: add --checkout/--no-checkout to update CE_NO_CHECKOUT bit

[Jeff King]

On Wed, Oct 01, 2008 at 11:04:05AM +0700, Nguyễn Thái Ngọc Duy wrote:

> +test_expect_success 'setup' '
> +	mkdir sub &&
> +	touch 1 2 sub/1 sub/2 &&
> +	git add 1 2 sub/1 sub/2

Mind-boggling, but this manages to break on Solaris. Fix is
below.

-- >8 --
t2104: touch portability fix

Some versions of touch support the syntax:

  touch [MMDDhhmm[yy]] file...

which makes the first argument an optional time
specification. They can get confused by

  touch 1 2 foo bar

as they assume that '1' is a bogus time specification. This
is broken, for example, with /bin/touch on Solaris 8.

To fix it, we'll just reverse the order of arguments so that
an unambiguous argument is in the slot for the optional time
specification.

Signed-off-by: Jeff King <peff@peff.net>
---
Note that this has implications for 'touch "$FOO" "$BAR"'
used in scripts if FOO might be entirely numeric. However, a
quick grep shows we usually touch one file at a time.

 t/t2104-update-index-no-checkout.sh |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/t/t2104-update-index-no-checkout.sh b/t/t2104-update-index-no-checkout.sh
index be9f913..37a6861 100755
--- a/t/t2104-update-index-no-checkout.sh
+++ b/t/t2104-update-index-no-checkout.sh
@@ -9,7 +9,7 @@ test_description='git update-index no-checkout bits (a.k.a sparse checkout)'
 
 test_expect_success 'setup' '
 	mkdir sub &&
-	touch 1 2 sub/1 sub/2 &&
+	touch sub/1 sub/2 1 2 &&
 	git add 1 2 sub/1 sub/2
 '
 
-- 
1.6.0.4.984.gbd0b7

Re: [PATCH 5/9] update-index: add --checkout/--no-checkout to update CE_NO_CHECKOUT bit

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Wed, Oct 01, 2008 at 11:04:05AM +0700, Nguyễn Thái Ngọc Duy wrote:
>
>> +test_expect_success 'setup' '
>> +	mkdir sub &&
>> +	touch 1 2 sub/1 sub/2 &&
>> +	git add 1 2 sub/1 sub/2
>
> Mind-boggling, but this manages to break on Solaris. Fix is
> below.

I tend to avoid "touch", not specifically for this reason, but surely this
is another reason why ">sub/1" is much better way to create a throw-away
file ;-)

> Note that this has implications for 'touch "$FOO" "$BAR"'
> used in scripts if FOO might be entirely numeric. However, a
> quick grep shows we usually touch one file at a time.

You can always do:

	: >>"$FOO"

Re: [PATCH 5/9] update-index: add --checkout/--no-checkout to update CE_NO_CHECKOUT bit

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Jeff King <peff@peff.net> writes:
>
>> On Wed, Oct 01, 2008 at 11:04:05AM +0700, Nguyễn Thái Ngọc Duy wrote:
>>
>>> +test_expect_success 'setup' '
>>> +	mkdir sub &&
>>> +	touch 1 2 sub/1 sub/2 &&
>>> +	git add 1 2 sub/1 sub/2
>>
>> Mind-boggling, but this manages to break on Solaris. Fix is
>> below.
>
> I tend to avoid "touch", not specifically for this reason, but surely this
> is another reason why ">sub/1" is much better way to create a throw-away
> file ;-)
>
>> Note that this has implications for 'touch "$FOO" "$BAR"'
>> used in scripts if FOO might be entirely numeric. However, a
>> quick grep shows we usually touch one file at a time.
>
> You can always do:
>
> 	: >>"$FOO"

Having said all that, I wouldn't suggest redoing the patch using >>
redirection.  But change from "touch 1 nondigit" to "touch nondigit 1"
is a bit too subtle to my taste.  Let's write it this way instead:

diff --git a/t/t2104-update-index-no-checkout.sh b/t/t2104-update-index-no-checkout.sh
index be9f913..37a6861 100755
--- a/t/t2104-update-index-no-checkout.sh
+++ b/t/t2104-update-index-no-checkout.sh
@@ -9,7 +9,7 @@ test_description='git update-index no-checkout bits (a.k.a sparse checkout)'
 
 test_expect_success 'setup' '
 	mkdir sub &&
-	touch 1 2 sub/1 sub/2 &&
+	touch ./1 ./2 sub/1 sub/2 &&
 	git add 1 2 sub/1 sub/2
 '
 

Re: [PATCH 5/9] update-index: add --checkout/--no-checkout to update CE_NO_CHECKOUT bit

[Jeff King]

On Tue, Nov 18, 2008 at 06:18:10PM -0800, Junio C Hamano wrote:

> Having said all that, I wouldn't suggest redoing the patch using >>
> redirection.  But change from "touch 1 nondigit" to "touch nondigit 1"
> is a bit too subtle to my taste.  Let's write it this way instead:

Yes, I also dislike the subtlety, but my "obvious" idea was something
like:

  for i in 1 2 sub/1 sub/2; do
    touch $i
  done

which just seemed clunky. But:

> -	touch 1 2 sub/1 sub/2 &&
> +	touch ./1 ./2 sub/1 sub/2 &&

this is less clunky, and I have confirmed that it solves the problem. I
just wasn't clever enough to think of it in the first place. ;)

-Peff

Re: [PATCH 5/9] update-index: add --checkout/--no-checkout to update CE_NO_CHECKOUT bit

[Nguyen Thai Ngoc Duy]

On 11/19/08, Jeff King <peff@peff.net> wrote:
> On Tue, Nov 18, 2008 at 06:18:10PM -0800, Junio C Hamano wrote:
>
>  > Having said all that, I wouldn't suggest redoing the patch using >>
>  > redirection.  But change from "touch 1 nondigit" to "touch nondigit 1"
>  > is a bit too subtle to my taste.  Let's write it this way instead:
>
>
> Yes, I also dislike the subtlety, but my "obvious" idea was something
>  like:
>
>   for i in 1 2 sub/1 sub/2; do
>     touch $i
>   done
>
>  which just seemed clunky. But:
>
>
>  > -     touch 1 2 sub/1 sub/2 &&
>  > +     touch ./1 ./2 sub/1 sub/2 &&
>
>
> this is less clunky, and I have confirmed that it solves the problem. I
>  just wasn't clever enough to think of it in the first place. ;)

Thanks for catching. The last half also has the same problem. Another
way is maybe just stay away for those numbers, naming the files by
alphabet. Just wonder if we could have some ways to automatically
catch this kind of bug in the future.

-- 
Duy

Re: [PATCH 5/9] update-index: add --checkout/--no-checkout to update CE_NO_CHECKOUT bit

[Jeff King]

On Thu, Nov 20, 2008 at 10:26:48PM +0700, Nguyen Thai Ngoc Duy wrote:

> Thanks for catching. The last half also has the same problem. Another

I'm not sure what you mean by "the last half also has the same problem";
with the patch I posted (or Junio's patch) the test works fine for me.

> way is maybe just stay away for those numbers, naming the files by
> alphabet. Just wonder if we could have some ways to automatically
> catch this kind of bug in the future.

Dscho suggested something similar. I would be happy if somebody wrote a
portability lint that found problems in shell constructs and calling
conventions of tools. In practice, though, I think it ends up being
quite hard to catalog all of the quirks of every platform (and
certainly, I would never have thought that this would break -- as it
was, after it _did_ break I had to sit scratching my head wondering how
that piece of code could be wrong).

So I think a simpler approach makes sense: write tests that make sure
the system is doing what you want, and then run those tests periodically
in the environments that you care about checking. When it breaks, you
know there is a problem. :) And that is exactly how this bug was caught.

-Peff

Re: [PATCH 5/9] update-index: add --checkout/--no-checkout to update CE_NO_CHECKOUT bit

[Nguyen Thai Ngoc Duy]

On 11/20/08, Jeff King <peff@peff.net> wrote:
> On Thu, Nov 20, 2008 at 10:26:48PM +0700, Nguyen Thai Ngoc Duy wrote:
>
>  > Thanks for catching. The last half also has the same problem. Another
>
>
> I'm not sure what you mean by "the last half also has the same problem";
>  with the patch I posted (or Junio's patch) the test works fine for me.

Sorry for the confusion. I meant the second half of the series,
implementing UI interface for sparce checkout, which is not in pu yet.
-- 
Duy

Re: [PATCH 5/9] update-index: add --checkout/--no-checkout to update CE_NO_CHECKOUT bit

[Junio C Hamano]

"Nguyen Thai Ngoc Duy" <pclouds@gmail.com> writes:

>> Yes, I also dislike the subtlety, but my "obvious" idea was something
>>  like:
>>
>>   for i in 1 2 sub/1 sub/2; do
>>     touch $i
>>   done
>>
>>  which just seemed clunky. But:
>>
>>
>>  > -     touch 1 2 sub/1 sub/2 &&
>>  > +     touch ./1 ./2 sub/1 sub/2 &&
>>
>>
>> this is less clunky, and I have confirmed that it solves the problem. I
>>  just wasn't clever enough to think of it in the first place. ;)
>
> Thanks for catching. The last half also has the same problem. Another
> way is maybe just stay away for those numbers, naming the files by
> ... Just wonder if we could have some ways to automatically
> catch this kind of bug in the future.

A tool to do so essentially amounts to coming up with a set of POSIX
command line tools that know and flag all the known bugs broken platform
tools have, at the same time producing a reasonable end result so that it
can continue running and detecting more portability issues in the scripts.

Using dash as the shell helps to catch use of constructs outside POSIX,
running various GNU tools with POSIXLY_CORRECT=YesPlease may have similar
effects, but quirks specific to particular platforms like the above is
fundamentally hard to check.  Running autobuilder farms like Jeff does is
probably the best thing we can do.

Re: [PATCH/resent 0/9] Sparse checkout (first half)

[Santi Béjar]

Not in this half but I've seen that you added a --default-sparse flag
to "git clone". I think it is useless as mostly always the one given
in -sparse-checkout should be used.

Thanks

Santi

On Wed, Oct 1, 2008 at 6:04 AM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
> This is the first half of the series, making git ready for sparse
> checkout. The only difference from the last (first half) sent
> series is safeguard bitmask fix in 1/9
>
> Nguyễn Thái Ngọc Duy (9):
>  Extend index to save more flags
>  Introduce CE_NO_CHECKOUT bit
>  ls-files: add options to support sparse checkout
>  update-index: refactor mark_valid() in preparation for new options
>  update-index: add --checkout/--no-checkout to update CE_NO_CHECKOUT
>    bit
>  ls-files: Add tests for --sparse and friends
>  Prevent diff machinery from examining worktree outside sparse
>    checkout
>  checkout_entry(): CE_NO_CHECKOUT on checked out entries.
>  grep: skip files outside sparse checkout area
>
>  .gitignore                            |    1 +
>  Documentation/git-checkout.txt        |   34 +++++++++++++++++
>  Documentation/git-grep.txt            |    4 +-
>  Documentation/git-ls-files.txt        |   24 +++++++++++-
>  Documentation/git-update-index.txt    |   13 ++++++
>  Makefile                              |    2 +-
>  builtin-grep.c                        |    7 +++-
>  builtin-ls-files.c                    |   41 ++++++++++++++++++--
>  builtin-update-index.c                |   40 ++++++++++++-------
>  cache.h                               |   66 +++++++++++++++++++++++++++++++--
>  diff-lib.c                            |    5 +-
>  diff.c                                |    4 +-
>  entry.c                               |    1 +
>  read-cache.c                          |   57 ++++++++++++++++++++++------
>  t/t2104-update-index-no-checkout.sh   |   36 ++++++++++++++++++
>  t/t3004-ls-files-sparse.sh            |   40 ++++++++++++++++++++
>  t/t3004/cached.expected               |    5 ++
>  t/t3004/deleted.expected              |    1 +
>  t/t3004/everything.expected           |   10 +++++
>  t/t3004/modified.expected             |    2 +
>  t/t3004/no-checkout.expected          |    2 +
>  t/t3004/orphaned-no-checkout.expected |    3 +
>  t/t3004/orphaned.expected             |    1 +
>  t/t3004/others.expected               |    2 +
>  t/t3004/sparse-cached.expected        |    3 +
>  t/t3004/sparse-everything.expected    |   11 +++++
>  test-index-version.c                  |   14 +++++++
>  27 files changed, 385 insertions(+), 44 deletions(-)
>  create mode 100755 t/t2104-update-index-no-checkout.sh
>  create mode 100755 t/t3004-ls-files-sparse.sh
>  create mode 100644 t/t3004/cached.expected
>  create mode 100644 t/t3004/deleted.expected
>  create mode 100644 t/t3004/everything.expected
>  create mode 100644 t/t3004/modified.expected
>  create mode 100644 t/t3004/no-checkout.expected
>  create mode 100644 t/t3004/orphaned-no-checkout.expected
>  create mode 100644 t/t3004/orphaned.expected
>  create mode 100644 t/t3004/others.expected
>  create mode 100644 t/t3004/sparse-cached.expected
>  create mode 100644 t/t3004/sparse-everything.expected
>  create mode 100644 test-index-version.c
>
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [PATCH/resent 0/9] Sparse checkout (first half)

[Santi Béjar]

On Wed, Oct 1, 2008 at 12:20 PM, Santi Béjar <santi@agolina.net> wrote:
> Not in this half but I've seen that you added a --default-sparse flag
> to "git clone". I think it is useless as mostly always the one given
> in -sparse-checkout should be used.

To be more precise, whenever you do "git clone --sparse-checkout" you
will want it to be the default sparse pattern.

Best regards,

Santi

Re: [PATCH/resent 0/9] Sparse checkout (first half)

[Nguyen Thai Ngoc Duy]

On 10/1/08, Santi Béjar <santi@agolina.net> wrote:
> On Wed, Oct 1, 2008 at 12:20 PM, Santi Béjar <santi@agolina.net> wrote:
>  > Not in this half but I've seen that you added a --default-sparse flag
>  > to "git clone". I think it is useless as mostly always the one given
>  > in -sparse-checkout should be used.
>
>
> To be more precise, whenever you do "git clone --sparse-checkout" you
>  will want it to be the default sparse pattern.

Yes, if only the default pattern gets updated properly when you update
your checkout area. If "git clone --sparse-checkout" makes the default
pattern, then "git checkout --reset-sparse" should reset the default
pattern as well. The hard part is how to update default pattern with
--include-sparse and --exclude-sparse. Also, people can use "git
update-index" to update checkout area, which should not touch default
pattern at all. Maybe just throw a warning when default pattern no
longer matches the checkout area, then let them decide.
-- 
Duy

Re: [PATCH/resent 0/9] Sparse checkout (first half)

[Santi Béjar]

On Wed, Oct 1, 2008 at 2:09 PM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
> On 10/1/08, Santi Béjar <santi@agolina.net> wrote:
>> On Wed, Oct 1, 2008 at 12:20 PM, Santi Béjar <santi@agolina.net> wrote:
>>  > Not in this half but I've seen that you added a --default-sparse flag
>>  > to "git clone". I think it is useless as mostly always the one given
>>  > in -sparse-checkout should be used.
>>
>>
>> To be more precise, whenever you do "git clone --sparse-checkout" you
>>  will want it to be the default sparse pattern.
>
> Yes, if only the default pattern gets updated properly when you update
> your checkout area.

I don't parse this sentence, but I cannot see the connection between
the default sparse pattern for _clone_ and updating it or the working
area.

> If "git clone --sparse-checkout" makes the default
> pattern, then "git checkout --reset-sparse" should reset the default
> pattern as well.

Makes sense.

> The hard part is how to update default pattern with
> --include-sparse and --exclude-sparse.

Maybe you could let the core.defaultsparse be defined multiple times.

[core]
        defaultsparse = Documentation/*
        defaultsparse = t/*
        defaultsparse = !t/*svn*

equivalent to

[core]
        defaultsparse = Documentation/*:t/*:!t/*svn*

but it moves the complexity to the parsing of the config.

> Also, people can use "git
> update-index" to update checkout area, which should not touch default
> pattern at all.

I would prefer having a --not-update-sparse-pattern.

The most common workflow should be the most straightforward. The most
common use-case would be somebody working _only_ in some subdirectory
(say Documentation), then what s/he normally does is:

1) cloning and default sparse chechout:
  $ git clone --sparse-checkout=Documentation/* ...
  or within an existing wd
  $ git checkout --reset-sparse=Documentation/* ...
  (maybe just --sparse?)

2) Work normally inside the Documentation directory...

3) Abort a merge with conflicts outside the sparse area:
  $ git reset --hard ORIG_HEAD)
  <Now there are files outside the default sparse area>
  $ git checkout --reset-sparse # with --sparse to set the sparse pattern

Including/excluding more paths in the default sparse checkout is not
something you would do normally, but I think it makes sense to add
them to the default pattern.

Please correct me if you think there are other use-cases, or more
things in my use-case.

> Maybe just throw a warning when default pattern no
> longer matches the checkout area, then let them decide.

Sure, and inform in the "git status" that you are in a sparse checkout.

Best regards,
Santi

Re: [PATCH/resent 0/9] Sparse checkout (first half)

[Santi Béjar]

On Thu, Oct 2, 2008 at 6:39 AM, Duy Nguyen <pclouds@gmail.com> wrote:
> On 10/01/2008 "Santi Béjar" <santi@agolina.net> wrote:
>>On Wed, Oct 1, 2008 at 2:09 PM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
>> > On 10/1/08, Santi Béjar <santi@agolina.net> wrote:
>> >> On Wed, Oct 1, 2008 at 12:20 PM, Santi Béjar <santi@agolina.net> wrote:
>> >>  > Not in this half but I've seen that you added a --default-sparse flag
>> >>  > to "git clone". I think it is useless as mostly always the one given
>> >>  > in -sparse-checkout should be used.
>> >>
>> >>
>> >> To be more precise, whenever you do "git clone --sparse-checkout" you
>> >>  will want it to be the default sparse pattern.
>> >
>> > Yes, if only the default pattern gets updated properly when you update
>> > your checkout area.
>>
>> I don't parse this sentence, but I cannot see the connection between
>> the default sparse pattern for _clone_ and updating it or the working
>> area.
>
> After clone with sparse "Documentation/" you may want to extend the checkout area to "t/" also. "git pull" after that should allow new files in Documentation/ as well as t/. So the default sparse pattern should be updated to "Documentation/:t/"

Sure.

[...]

>> > The hard part is how to update default pattern with
>> > --include-sparse and --exclude-sparse.
>>
>> Maybe you could let the core.defaultsparse be defined multiple times.
>>
>> [core]
>>         defaultsparse = Documentation/*
>>         defaultsparse = t/*
>>         defaultsparse = !t/*svn*
>>
>> equivalent to
>>
>> [core]
>>         defaultsparse = Documentation/*:t/*:!t/*svn*
>>
>> but it moves the complexity to the parsing of the config.
>
> There are two issues regarding --{in,ex}clude-sparse:
>  1. we need to save the prefix somehow. "git checkout --include-sparse='*.c'" in compat/ and the same command at top-level workdir give different result.

then, core.defaultsparse="compat//*.c" and core.defaultsparse="*.c", isn't it?

>  If we don't save prefix, we get wrong result with defaultsparse. This one is easy although it may make sparse patterns and .git{ignore,attributes} look different.

Possible bug, currently in your sparse-checkout branch the prefix is not saved:

$ cd git
$ cd Documentation
$ git checkout --sparse=RelNotes*
$ git config -l | grep sparse
core.defaultsparse=RelNotes-1.*

>  2. cruft gets accumulated in core.defaultsparse. Consider this (assume we cloned with sparse "Documentation/")
>
> for i in 1 2 3;do
> git checkout --include-sparse="t/"
> git checkout --exclude-sparse="t/" # oops, I did not mean to do that
> done
>
> The default sparse pattern would be "Documentation/:t/:!t/:t/:!t/:t/:!t/" while it should be just "Documentation/". Another example is
>
> git checkout --include-sparse=t/test-lib.sh
> git checkout --include-sparse=t/
>
> The defaultsparse would be "Documentation/:t/test-lib.sh:t/" while a better defaultsparse should be "Documentation/:t/"
>

But these are the steps to reproduce what the user asked for, so I
don't see a problem.

> But maybe I just overlook this. Maybe just show the defaultsparse to people and let them clean it with "--reset-sparse".

Sure.

>
>> > Also, people can use "git
>> > update-index" to update checkout area, which should not touch default
>> > pattern at all.
>>
>> I would prefer having a --not-update-sparse-pattern.
>
> "git update-index" does not understand "sparse patterns" at all. It operates at file level. But it is plumbling anyway so those who use it should take care of default sparse pattern themselves.

Well, "git checkout <paths>" does the trick.

Santi