[PATCH 0/6] Teach fetch/pull the on-demand mode and make it the default

[PATCH 0/6] Teach fetch/pull the on-demand mode and make it the default

[Jens Lehmann]

So here is my patch series to implement the new default "on-demand"
fetch mode and let "git submodule update" only fetch submodules when
the commit to check out is not already present in the submodule.
This is only done for populated submodules and prepares them for
disconnected operation without having to think about doing a
recursive fetch before going on a plane and having to add the '-N'
option to "git submodule update" while on it. Also merging submodule
commits in the superproject and "git diff --submodule" (which is
used by git gui and gitk) depend on the presence of those commits
referenced by the superproject to work.

The last commit is slightly orthogonal as it would be useful without
on-demand fetch too. With it rewinding a submodule to a commit that
is already present (e.g. because it was checked out earlier) will be
done without issuing a fetch even in current git. Together with the
other patches "git submodule update" won't have to fetch a submodule
at all anymore (at least unless the new "on-demand" default is
disabled through the also added configuration options).

Changes since the last version:
(see http://article.gmane.org/gmane.comp.version-control.git/163429)

*) Added the "on-demand" value to the "--recurse-submodules" command
   line option and the "submodule.<name>.fetchRecurseSubmodules" and
   "fetch.recurseSubmodules" config options.
*) The fetch is only done when the recorded submodule commit isn't
   already present.
*) Added test cases.

There are two things that could be optimized, but that can be the
topic of other patches:

*) The submodule fetches could be done in parallel.
*) It might be expensive to compute the newly fetched commits. If
   the transport helper could be asked for the list commits it just
   fetched that would go away.

I tend to think that this is suited for 1.7.5 but don't have any
objections against holding it back until 1.8.0 either. What do
others think?

Jens Lehmann (6):
  fetch/pull: recurse into submodules when necessary
  fetch/pull: Add the 'on-demand' value to the --recurse-submodules
    option
  config: teach the fetch.recurseSubmodules option the 'on-demand'
    value
  Submodules: Add 'on-demand' value for the 'fetchRecurseSubmodule'
    option
  fetch/pull: Don't recurse into a submodule when commits are already
    present
  submodule update: Don't fetch when the submodule commit is already
    present

 Documentation/config.txt        |   12 ++-
 Documentation/fetch-options.txt |   24 +++-
 Documentation/git-pull.txt      |    2 +-
 Documentation/gitmodules.txt    |    4 +-
 builtin/fetch.c                 |   49 ++++++--
 git-pull.sh                     |    3 +
 git-submodule.sh                |    2 +-
 submodule.c                     |  124 +++++++++++++++++--
 submodule.h                     |   11 ++-
 t/t5526-fetch-submodules.sh     |  255 +++++++++++++++++++++++++++++++++++++++
 t/t7403-submodule-sync.sh       |    2 +-
 t/t7406-submodule-update.sh     |   20 +++
 12 files changed, 471 insertions(+), 37 deletions(-)

-- 
1.7.4.1.190.g13e20

[PATCH 1/6] fetch/pull: recurse into submodules when necessary

[Jens Lehmann]

To be able to access all commits of populated submodules referenced by the
superproject it is sufficient to only then let "git fetch" recurse into a
submodule when the new commits fetched in the superproject record new
commits for it. Having these commits present is extremely useful when
using the "--submodule" option to "git diff" (which is what "git gui" and
"gitk" do since 1.6.6), as all submodule commits needed for creating a
descriptive output can be accessed. Also merging submodule commits (added
in 1.7.3) depends on the submodule commits in question being present to
work. Last but not least this enables disconnected operation when using
submodules, as all commits necessary for a successful "git submodule
update -N" will have been fetched automatically. So we choose this mode as
the default for fetch and pull.

Before a new or changed ref from upstream is updated in update_local_ref()
"git rev-list <new-sha1> --not --branches --remotes" is used to determine
all newly fetched commits. These are then walked and diffed against their
parent(s) to see if a submodule has been changed. If that is the case, its
path is stored to be fetched after the superproject fetch is completed.

Using the "--recurse-submodules" or the "--no-recurse-submodules" option
disables the examination of the fetched refs because the result will be
ignored anyway.

There is currently no infrastructure for storing deleted and new
submodules in the .git directory of the superproject. Thats why fetch and
pull for now only fetch submodules that are already checked out and are
not renamed.

In t7403 the "--no-recurse-submodules" argument had to be added to "git
pull" to avoid failure because of the moved upstream submodule repo.

Thanks-To: Jonathan Nieder <jrnieder@gmail.com>
Thanks-To: Heiko Voigt <hvoigt@hvoigt.net>
Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
---
 Documentation/fetch-options.txt |    8 +++
 builtin/fetch.c                 |   25 ++++++---
 submodule.c                     |   98 +++++++++++++++++++++++++++++++++--
 submodule.h                     |    9 +++
 t/t5526-fetch-submodules.sh     |  109 +++++++++++++++++++++++++++++++++++++++
 t/t7403-submodule-sync.sh       |    2 +-
 6 files changed, 235 insertions(+), 16 deletions(-)

diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt
index f37276e..1d770d9 100644
--- a/Documentation/fetch-options.txt
+++ b/Documentation/fetch-options.txt
@@ -73,6 +73,14 @@ ifndef::git-pull[]
 	Prepend <path> to paths printed in informative messages
 	such as "Fetching submodule foo".  This option is used
 	internally when recursing over submodules.
+
+--submodule-default=[yes|on-demand]::
+	This option is used internally to set the submodule recursion default
+	to either a boolean configuration value representing "true" (for
+	unconditonal recursion) or to "on-demand" (when only those submodules
+	should be fetched of which new commits have been fetched in its
+	superproject).
+	Using the "--[no-]recurse-submodules" option ignores this setting.
 endif::git-pull[]

 -u::
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 7efecfe..feb9392 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -28,12 +28,6 @@ enum {
 	TAGS_SET = 2
 };

-enum {
-	RECURSE_SUBMODULES_OFF = 0,
-	RECURSE_SUBMODULES_DEFAULT = 1,
-	RECURSE_SUBMODULES_ON = 2
-};
-
 static int all, append, dry_run, force, keep, multiple, prune, update_head_ok, verbosity;
 static int progress, recurse_submodules = RECURSE_SUBMODULES_DEFAULT;
 static int tags = TAGS_DEFAULT;
@@ -42,6 +36,7 @@ static const char *upload_pack;
 static struct strbuf default_rla = STRBUF_INIT;
 static struct transport *transport;
 static const char *submodule_prefix = "";
+static const char *submodule_default;

 static struct option builtin_fetch_options[] = {
 	OPT__VERBOSITY(&verbosity),
@@ -73,6 +68,8 @@ static struct option builtin_fetch_options[] = {
 		   "deepen history of shallow clone"),
 	{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, "dir",
 		   "prepend this to submodule path output", PARSE_OPT_HIDDEN },
+	{ OPTION_STRING, 0, "submodule-default", &submodule_default, NULL,
+		   "default mode for recursion", PARSE_OPT_HIDDEN },
 	OPT_END()
 };

@@ -284,6 +281,9 @@ static int update_local_ref(struct ref *ref,
 		else {
 			msg = "storing head";
 			what = "[new branch]";
+			if ((recurse_submodules != RECURSE_SUBMODULES_OFF) &&
+			    (recurse_submodules != RECURSE_SUBMODULES_ON))
+				check_for_new_submodule_commits(ref->new_sha1);
 		}

 		r = s_update_ref(msg, ref, 0);
@@ -299,6 +299,9 @@ static int update_local_ref(struct ref *ref,
 		strcpy(quickref, find_unique_abbrev(current->object.sha1, DEFAULT_ABBREV));
 		strcat(quickref, "..");
 		strcat(quickref, find_unique_abbrev(ref->new_sha1, DEFAULT_ABBREV));
+		if ((recurse_submodules != RECURSE_SUBMODULES_OFF) &&
+		    (recurse_submodules != RECURSE_SUBMODULES_ON))
+			check_for_new_submodule_commits(ref->new_sha1);
 		r = s_update_ref("fast-forward", ref, 1);
 		sprintf(display, "%c %-*s %-*s -> %s%s", r ? '!' : ' ',
 			TRANSPORT_SUMMARY_WIDTH, quickref, REFCOL_WIDTH, remote,
@@ -310,6 +313,9 @@ static int update_local_ref(struct ref *ref,
 		strcpy(quickref, find_unique_abbrev(current->object.sha1, DEFAULT_ABBREV));
 		strcat(quickref, "...");
 		strcat(quickref, find_unique_abbrev(ref->new_sha1, DEFAULT_ABBREV));
+		if ((recurse_submodules != RECURSE_SUBMODULES_OFF) &&
+		    (recurse_submodules != RECURSE_SUBMODULES_ON))
+			check_for_new_submodule_commits(ref->new_sha1);
 		r = s_update_ref("forced-update", ref, 1);
 		sprintf(display, "%c %-*s %-*s -> %s  (%s)", r ? '!' : '+',
 			TRANSPORT_SUMMARY_WIDTH, quickref, REFCOL_WIDTH, remote,
@@ -949,9 +955,10 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	if (!result && (recurse_submodules != RECURSE_SUBMODULES_OFF)) {
 		const char *options[10];
 		int num_options = 0;
-		/* Set recursion as default when we already are recursing */
-		if (submodule_prefix[0])
-			set_config_fetch_recurse_submodules(1);
+		if (submodule_default) {
+			int arg = parse_fetch_recurse_submodules_arg(submodule_default);
+			set_config_fetch_recurse_submodules(arg);
+		}
 		gitmodules_config();
 		git_config(submodule_config, NULL);
 		add_options_to_argv(&num_options, options);
diff --git a/submodule.c b/submodule.c
index 6f1c107..c8c3a99 100644
--- a/submodule.c
+++ b/submodule.c
@@ -12,7 +12,8 @@
 struct string_list config_name_for_path;
 struct string_list config_fetch_recurse_submodules_for_name;
 struct string_list config_ignore_for_name;
-static int config_fetch_recurse_submodules;
+static int config_fetch_recurse_submodules = RECURSE_SUBMODULES_ON_DEMAND;
+struct string_list changed_submodule_paths;

 static int add_submodule_odb(const char *path)
 {
@@ -152,6 +153,20 @@ void handle_ignore_submodules_arg(struct diff_options *diffopt,
 		die("bad --ignore-submodules argument: %s", arg);
 }

+int parse_fetch_recurse_submodules_arg(const char *arg)
+{
+	switch (git_config_maybe_bool("", arg)) {
+	case 1:
+		return RECURSE_SUBMODULES_ON;
+	case 0:
+		return RECURSE_SUBMODULES_OFF;
+	default:
+		if (!strcmp(arg, "on-demand"))
+			return RECURSE_SUBMODULES_ON_DEMAND;
+		die("bad --recurse-submodules argument: %s", arg);
+	}
+}
+
 void show_submodule_summary(FILE *f, const char *path,
 		unsigned char one[20], unsigned char two[20],
 		unsigned dirty_submodule,
@@ -248,11 +263,73 @@ void set_config_fetch_recurse_submodules(int value)
 	config_fetch_recurse_submodules = value;
 }

+static void submodule_collect_changed_cb(struct diff_queue_struct *q,
+					 struct diff_options *options,
+					 void *data)
+{
+	int i;
+	for (i = 0; i < q->nr; i++) {
+		struct diff_filepair *p = q->queue[i];
+		if (!S_ISGITLINK(p->two->mode))
+			continue;
+
+		if (S_ISGITLINK(p->one->mode)) {
+			/* NEEDSWORK: We should honor the name configured in
+			 * the .gitmodules file of the commit we are examining
+			 * here to be able to correctly follow submodules
+			 * being moved around. */
+			struct string_list_item *path;
+			path = unsorted_string_list_lookup(&changed_submodule_paths, p->two->path);
+			if (!path)
+				string_list_append(&changed_submodule_paths, xstrdup(p->two->path));
+		} else {
+			/* Submodule is new or was moved here */
+			/* NEEDSWORK: When the .git directories of submodules
+			 * live inside the superprojects .git directory some
+			 * day we should fetch new submodules directly into
+			 * that location too when config or options request
+			 * that so they can be checked out from there. */
+			continue;
+		}
+	}
+}
+
+void check_for_new_submodule_commits(unsigned char new_sha1[20])
+{
+	struct rev_info rev;
+	struct commit *commit;
+	int argc = 5;
+	const char *argv[] = {NULL, NULL, "--not", "--branches", "--remotes", NULL};
+
+	init_revisions(&rev, NULL);
+	argv[1] = xstrdup(sha1_to_hex(new_sha1));
+	setup_revisions(argc, argv, &rev, NULL);
+	if (prepare_revision_walk(&rev))
+		die("revision walk setup failed");
+
+	while ((commit = get_revision(&rev))) {
+		struct commit_list *parent = commit->parents;
+		while (parent) {
+			struct diff_options diff_opts;
+			diff_setup(&diff_opts);
+			diff_opts.output_format |= DIFF_FORMAT_CALLBACK;
+			diff_opts.format_callback = submodule_collect_changed_cb;
+			if (diff_setup_done(&diff_opts) < 0)
+				die("diff_setup_done failed");
+			diff_tree_sha1(parent->item->object.sha1, commit->object.sha1, "", &diff_opts);
+			diffcore_std(&diff_opts);
+			diff_flush(&diff_opts);
+			parent = parent->next;
+		}
+	}
+	free((char *)argv[1]);
+}
+
 int fetch_populated_submodules(int num_options, const char **options,
 			       const char *prefix, int ignore_config,
 			       int quiet)
 {
-	int i, result = 0, argc = 0;
+	int i, result = 0, argc = 0, default_argc;
 	struct child_process cp;
 	const char **argv;
 	struct string_list_item *name_for_path;
@@ -264,11 +341,13 @@ int fetch_populated_submodules(int num_options, const char **options,
 		if (read_cache() < 0)
 			die("index file corrupt");

-	/* 4: "fetch" (options) "--submodule-prefix" prefix NULL */
-	argv = xcalloc(num_options + 4, sizeof(const char *));
+	/* 6: "fetch" (options) --submodule-default default "--submodule-prefix" prefix NULL */
+	argv = xcalloc(num_options + 6, sizeof(const char *));
 	argv[argc++] = "fetch";
 	for (i = 0; i < num_options; i++)
 		argv[argc++] = options[i];
+	argv[argc++] = "--submodule-default";
+	default_argc = argc++;
 	argv[argc++] = "--submodule-prefix";

 	memset(&cp, 0, sizeof(cp));
@@ -282,7 +361,7 @@ int fetch_populated_submodules(int num_options, const char **options,
 		struct strbuf submodule_git_dir = STRBUF_INIT;
 		struct strbuf submodule_prefix = STRBUF_INIT;
 		struct cache_entry *ce = active_cache[i];
-		const char *git_dir, *name;
+		const char *git_dir, *name, *default_argv;

 		if (!S_ISGITLINK(ce->ce_mode))
 			continue;
@@ -292,6 +371,7 @@ int fetch_populated_submodules(int num_options, const char **options,
 		if (name_for_path)
 			name = name_for_path->util;

+		default_argv = "yes";
 		if (!ignore_config) {
 			struct string_list_item *fetch_recurse_submodules_option;
 			fetch_recurse_submodules_option = unsorted_string_list_lookup(&config_fetch_recurse_submodules_for_name, name);
@@ -299,8 +379,13 @@ int fetch_populated_submodules(int num_options, const char **options,
 				if (!fetch_recurse_submodules_option->util)
 					continue;
 			} else {
-				if (!config_fetch_recurse_submodules)
+				if (config_fetch_recurse_submodules == RECURSE_SUBMODULES_OFF)
 					continue;
+				if (config_fetch_recurse_submodules == RECURSE_SUBMODULES_ON_DEMAND) {
+					if (!unsorted_string_list_lookup(&changed_submodule_paths, ce->name))
+						continue;
+					default_argv = "on-demand";
+				}
 			}
 		}

@@ -314,6 +399,7 @@ int fetch_populated_submodules(int num_options, const char **options,
 			if (!quiet)
 				printf("Fetching submodule %s%s\n", prefix, ce->name);
 			cp.dir = submodule_path.buf;
+			argv[default_argc] = default_argv;
 			argv[argc] = submodule_prefix.buf;
 			if (run_command(&cp))
 				result = 1;
diff --git a/submodule.h b/submodule.h
index 4729023..aca023a 100644
--- a/submodule.h
+++ b/submodule.h
@@ -3,17 +3,26 @@

 struct diff_options;

+enum {
+	RECURSE_SUBMODULES_ON_DEMAND = -1,
+	RECURSE_SUBMODULES_OFF = 0,
+	RECURSE_SUBMODULES_DEFAULT = 1,
+	RECURSE_SUBMODULES_ON = 2
+};
+
 void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,
 		const char *path);
 int submodule_config(const char *var, const char *value, void *cb);
 void gitmodules_config();
 int parse_submodule_config_option(const char *var, const char *value);
 void handle_ignore_submodules_arg(struct diff_options *diffopt, const char *);
+int parse_fetch_recurse_submodules_arg(const char *arg);
 void show_submodule_summary(FILE *f, const char *path,
 		unsigned char one[20], unsigned char two[20],
 		unsigned dirty_submodule,
 		const char *del, const char *add, const char *reset);
 void set_config_fetch_recurse_submodules(int value);
+void check_for_new_submodule_commits(unsigned char new_sha1[20]);
 int fetch_populated_submodules(int num_options, const char **options,
 			       const char *prefix, int ignore_config,
 			       int quiet);
diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh
index a5f4585..6d92f7a 100755
--- a/t/t5526-fetch-submodules.sh
+++ b/t/t5526-fetch-submodules.sh
@@ -192,4 +192,113 @@ test_expect_success "--no-recurse-submodules overrides config setting" '
 	! test -s actual.err
 '

+test_expect_success "Recursion doesn't happen when no new commits are fetched in the superproject" '
+	(
+		cd downstream &&
+		(
+			cd submodule &&
+			git config --unset fetch.recurseSubmodules
+		) &&
+		git config --unset fetch.recurseSubmodules
+		git fetch >../actual.out 2>../actual.err
+	) &&
+	! test -s actual.out &&
+	! test -s actual.err
+'
+
+test_expect_success "Recursion stops when no new submodule commits are fetched" '
+	head1=$(git rev-parse --short HEAD) &&
+	git add submodule &&
+	git commit -m "new submodule" &&
+	head2=$(git rev-parse --short HEAD) &&
+	echo "Fetching submodule submodule" > expect.out.sub &&
+	echo "From $pwd/." > expect.err.sub &&
+	echo "   $head1..$head2  master     -> origin/master" >> expect.err.sub
+	head -2 expect.err >> expect.err.sub &&
+	(
+		cd downstream &&
+		git fetch >../actual.out 2>../actual.err
+	) &&
+	test_cmp expect.err.sub actual.err &&
+	test_cmp expect.out.sub actual.out
+'
+
+test_expect_success "Recursion doesn't happen when new superproject commits don't change any submodules" '
+	add_upstream_commit &&
+	head1=$(git rev-parse --short HEAD) &&
+	echo a > file &&
+	git add file &&
+	git commit -m "new file" &&
+	head2=$(git rev-parse --short HEAD) &&
+	echo "From $pwd/." > expect.err.file &&
+	echo "   $head1..$head2  master     -> origin/master" >> expect.err.file &&
+	(
+		cd downstream &&
+		git fetch >../actual.out 2>../actual.err
+	) &&
+	! test -s actual.out &&
+	test_cmp expect.err.file actual.err
+'
+
+test_expect_success "Recursion picks up config in submodule" '
+	(
+		cd downstream &&
+		git fetch --recurse-submodules &&
+		(
+			cd submodule &&
+			git config fetch.recurseSubmodules true
+		)
+	) &&
+	add_upstream_commit &&
+	head1=$(git rev-parse --short HEAD) &&
+	git add submodule &&
+	git commit -m "new submodule" &&
+	head2=$(git rev-parse --short HEAD) &&
+	echo "From $pwd/." > expect.err.sub &&
+	echo "   $head1..$head2  master     -> origin/master" >> expect.err.sub &&
+	cat expect.err >> expect.err.sub &&
+	(
+		cd downstream &&
+		git fetch >../actual.out 2>../actual.err &&
+		(
+			cd submodule &&
+			git config --unset fetch.recurseSubmodules
+		)
+	) &&
+	test_cmp expect.err.sub actual.err &&
+	test_cmp expect.out actual.out
+'
+
+test_expect_success "Recursion picks up all submodules when necessary" '
+	add_upstream_commit &&
+	(
+		cd submodule &&
+		(
+			cd deepsubmodule &&
+			git fetch &&
+			git checkout -q FETCH_HEAD
+		) &&
+		head1=$(git rev-parse --short HEAD^) &&
+		git add deepsubmodule &&
+		git commit -m "new deepsubmodule"
+		head2=$(git rev-parse --short HEAD) &&
+		echo "From $pwd/submodule" > ../expect.err.sub &&
+		echo "   $head1..$head2  master     -> origin/master" >> ../expect.err.sub
+	) &&
+	head1=$(git rev-parse --short HEAD) &&
+	git add submodule &&
+	git commit -m "new submodule" &&
+	head2=$(git rev-parse --short HEAD) &&
+	echo "From $pwd/." > expect.err.2 &&
+	echo "   $head1..$head2  master     -> origin/master" >> expect.err.2 &&
+	cat expect.err.sub >> expect.err.2 &&
+	tail -2 expect.err >> expect.err.2 &&
+	(
+		cd downstream &&
+		git fetch >../actual.out 2>../actual.err
+	) &&
+	test_cmp expect.err.2 actual.err &&
+	test_cmp expect.out actual.out
+'
+
 test_done
diff --git a/t/t7403-submodule-sync.sh b/t/t7403-submodule-sync.sh
index e5b1953..d600583 100755
--- a/t/t7403-submodule-sync.sh
+++ b/t/t7403-submodule-sync.sh
@@ -52,7 +52,7 @@ test_expect_success 'change submodule url' '

 test_expect_success '"git submodule sync" should update submodule URLs' '
 	(cd super-clone &&
-	 git pull &&
+	 git pull --no-recurse-submodules &&
 	 git submodule sync
 	) &&
 	test -d "$(git config -f super-clone/submodule/.git/config \
-- 
1.7.4.1.190.g13e20

[PATCH 2/6] fetch/pull: Add the 'on-demand' value to the --recurse-submodules option

[Jens Lehmann]

Until now the --recurse-submodules option could only be used to either
fetch all populated submodules recursively or to disable recursion
completely. As fetch and pull now by default just fetch those submodules
for which new commits have been fetched in the superproject, a command
line option to enforce that behavior is needed to be able to override
configuration settings.

Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
---
 Documentation/fetch-options.txt |   16 +++++++--
 Documentation/git-pull.txt      |    2 +-
 builtin/fetch.c                 |   24 +++++++++++--
 git-pull.sh                     |    3 ++
 submodule.c                     |    8 +++-
 submodule.h                     |    2 +-
 t/t5526-fetch-submodules.sh     |   71 +++++++++++++++++++++++++++++++++++++++
 7 files changed, 115 insertions(+), 11 deletions(-)

diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt
index 1d770d9..3fbdb53 100644
--- a/Documentation/fetch-options.txt
+++ b/Documentation/fetch-options.txt
@@ -65,9 +65,19 @@ ifndef::git-pull[]
 	specified with the remote.<name>.tagopt setting. See
 	linkgit:git-config[1].

---[no-]recurse-submodules::
-	This option controls if new commits of all populated submodules should
-	be fetched too (see linkgit:git-config[1] and linkgit:gitmodules[5]).
+--recurse-submodules[=yes|on-demand|no]::
+	This option controls if and under what conditions new commits of all
+	populated submodules should be fetched too. It can be used as a
+	boolean option to completely disable recursion when set to 'no' or to
+	unconditionally recurse into all populated submodules when set to
+	'yes', which is the default when this option is used without any
+	value. If 'on-demand' is used, it will only recurse into those
+	submodules where new commits have been fetched in the superproject
+	(also see linkgit:git-config[1] and linkgit:gitmodules[5]).
+
+--no-recurse-submodules::
+	Disable recursive fetching of submodules (this has the same effect as
+	using the '--recurse-submodules=no' option).

 --submodule-prefix=<path>::
 	Prepend <path> to paths printed in informative messages
diff --git a/Documentation/git-pull.txt b/Documentation/git-pull.txt
index b33e6be..c45efb3 100644
--- a/Documentation/git-pull.txt
+++ b/Documentation/git-pull.txt
@@ -84,7 +84,7 @@ must be given before the options meant for 'git fetch'.
 --verbose::
 	Pass --verbose to git-fetch and git-merge.

---[no-]recurse-submodules::
+--[no-]recurse-submodules[=yes|on-demand|no]::
 	This option controls if new commits of all populated submodules should
 	be fetched too (see linkgit:git-config[1] and linkgit:gitmodules[5]).
 	That might be necessary to get the data needed for merging submodule
diff --git a/builtin/fetch.c b/builtin/fetch.c
index feb9392..3743b01 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -38,6 +38,20 @@ static struct transport *transport;
 static const char *submodule_prefix = "";
 static const char *submodule_default;

+static int option_parse_recurse_submodules(const struct option *opt,
+				   const char *arg, int unset)
+{
+	if (unset) {
+		recurse_submodules = RECURSE_SUBMODULES_OFF;
+	} else {
+		if (arg)
+			recurse_submodules = parse_fetch_recurse_submodules_arg(arg);
+		else
+			recurse_submodules = RECURSE_SUBMODULES_ON;
+	}
+	return 0;
+}
+
 static struct option builtin_fetch_options[] = {
 	OPT__VERBOSITY(&verbosity),
 	OPT_BOOLEAN(0, "all", &all,
@@ -55,9 +69,9 @@ static struct option builtin_fetch_options[] = {
 		    "do not fetch all tags (--no-tags)", TAGS_UNSET),
 	OPT_BOOLEAN('p', "prune", &prune,
 		    "prune remote-tracking branches no longer on remote"),
-	OPT_SET_INT(0, "recurse-submodules", &recurse_submodules,
-		    "control recursive fetching of submodules",
-		    RECURSE_SUBMODULES_ON),
+	{ OPTION_CALLBACK, 0, "recurse-submodules", NULL, "on-demand",
+		"control recursive fetching of submodules",
+		PARSE_OPT_OPTARG, option_parse_recurse_submodules },
 	OPT_BOOLEAN(0, "dry-run", &dry_run,
 		    "dry run"),
 	OPT_BOOLEAN('k', "keep", &keep, "keep downloaded pack"),
@@ -816,6 +830,8 @@ static void add_options_to_argv(int *argc, const char **argv)
 		argv[(*argc)++] = "--keep";
 	if (recurse_submodules == RECURSE_SUBMODULES_ON)
 		argv[(*argc)++] = "--recurse-submodules";
+	else if (recurse_submodules == RECURSE_SUBMODULES_ON_DEMAND)
+		argv[(*argc)++] = "--recurse-submodules=on-demand";
 	if (verbosity >= 2)
 		argv[(*argc)++] = "-v";
 	if (verbosity >= 1)
@@ -964,7 +980,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 		add_options_to_argv(&num_options, options);
 		result = fetch_populated_submodules(num_options, options,
 						    submodule_prefix,
-						    recurse_submodules == RECURSE_SUBMODULES_ON,
+						    recurse_submodules,
 						    verbosity < 0);
 	}

diff --git a/git-pull.sh b/git-pull.sh
index 86517e9..7a97d8e 100755
--- a/git-pull.sh
+++ b/git-pull.sh
@@ -108,6 +108,9 @@ do
 	--recurse-submodules)
 		recurse_submodules=--recurse-submodules
 		;;
+	--recurse-submodules=*)
+		recurse_submodules="$1"
+		;;
 	--no-recurse-submodules)
 		recurse_submodules=--no-recurse-submodules
 		;;
diff --git a/submodule.c b/submodule.c
index c8c3a99..398616a 100644
--- a/submodule.c
+++ b/submodule.c
@@ -326,7 +326,7 @@ void check_for_new_submodule_commits(unsigned char new_sha1[20])
 }

 int fetch_populated_submodules(int num_options, const char **options,
-			       const char *prefix, int ignore_config,
+			       const char *prefix, int command_line_option,
 			       int quiet)
 {
 	int i, result = 0, argc = 0, default_argc;
@@ -372,7 +372,7 @@ int fetch_populated_submodules(int num_options, const char **options,
 			name = name_for_path->util;

 		default_argv = "yes";
-		if (!ignore_config) {
+		if (command_line_option == RECURSE_SUBMODULES_DEFAULT) {
 			struct string_list_item *fetch_recurse_submodules_option;
 			fetch_recurse_submodules_option = unsorted_string_list_lookup(&config_fetch_recurse_submodules_for_name, name);
 			if (fetch_recurse_submodules_option) {
@@ -387,6 +387,10 @@ int fetch_populated_submodules(int num_options, const char **options,
 					default_argv = "on-demand";
 				}
 			}
+		} else if (command_line_option == RECURSE_SUBMODULES_ON_DEMAND) {
+			if (!unsorted_string_list_lookup(&changed_submodule_paths, ce->name))
+				continue;
+			default_argv = "on-demand";
 		}

 		strbuf_addf(&submodule_path, "%s/%s", work_tree, ce->name);
diff --git a/submodule.h b/submodule.h
index aca023a..974cc86 100644
--- a/submodule.h
+++ b/submodule.h
@@ -24,7 +24,7 @@ void show_submodule_summary(FILE *f, const char *path,
 void set_config_fetch_recurse_submodules(int value);
 void check_for_new_submodule_commits(unsigned char new_sha1[20]);
 int fetch_populated_submodules(int num_options, const char **options,
-			       const char *prefix, int ignore_config,
+			       const char *prefix, int command_line_option,
 			       int quiet);
 unsigned is_submodule_modified(const char *path, int ignore_untracked);
 int merge_submodule(unsigned char result[20], const char *path, const unsigned char base[20],
diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh
index 6d92f7a..4cd723c 100755
--- a/t/t5526-fetch-submodules.sh
+++ b/t/t5526-fetch-submodules.sh
@@ -301,4 +301,75 @@ test_expect_success "Recursion picks up all submodules when necessary" '
 	test_cmp expect.out actual.out
 '

+test_expect_success "'--recurse-submodules=on-demand' doesn't recurse when no new commits are fetched in the superproject (and ignores config)" '
+	add_upstream_commit &&
+	(
+		cd submodule &&
+		(
+			cd deepsubmodule &&
+			git fetch &&
+			git checkout -q FETCH_HEAD
+		) &&
+		head1=$(git rev-parse --short HEAD^) &&
+		git add deepsubmodule &&
+		git commit -m "new deepsubmodule"
+		head2=$(git rev-parse --short HEAD) &&
+		echo "From $pwd/submodule" > ../expect.err.sub &&
+		echo "   $head1..$head2  master     -> origin/master" >> ../expect.err.sub
+	) &&
+	(
+		cd downstream &&
+		git config fetch.recurseSubmodules true &&
+		git fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err &&
+		git config --unset fetch.recurseSubmodules
+	) &&
+	! test -s actual.out &&
+	! test -s actual.err
+'
+
+test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)" '
+	head1=$(git rev-parse --short HEAD) &&
+	git add submodule &&
+	git commit -m "new submodule" &&
+	head2=$(git rev-parse --short HEAD) &&
+	tail -2 expect.err > expect.err.deepsub &&
+	echo "From $pwd/." > expect.err &&
+	echo "   $head1..$head2  master     -> origin/master" >> expect.err
+	cat expect.err.sub >> expect.err &&
+	cat expect.err.deepsub >> expect.err &&
+	(
+		cd downstream &&
+		git config fetch.recurseSubmodules false &&
+		(
+			cd submodule &&
+			git config -f .gitmodules submodule.deepsubmodule.fetchRecursive false
+		) &&
+		git fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err &&
+		git config --unset fetch.recurseSubmodules
+		(
+			cd submodule &&
+			git config --unset -f .gitmodules submodule.deepsubmodule.fetchRecursive
+		)
+	) &&
+	test_cmp expect.out actual.out &&
+	test_cmp expect.err actual.err
+'
+
+test_expect_success "'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)" '
+	add_upstream_commit &&
+	head1=$(git rev-parse --short HEAD) &&
+	echo a >> file &&
+	git add file &&
+	git commit -m "new file" &&
+	head2=$(git rev-parse --short HEAD) &&
+	echo "From $pwd/." > expect.err.file &&
+	echo "   $head1..$head2  master     -> origin/master" >> expect.err.file &&
+	(
+		cd downstream &&
+		git fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err
+	) &&
+	! test -s actual.out &&
+	test_cmp expect.err.file actual.err
+'
+
 test_done
-- 
1.7.4.1.190.g13e20

[PATCH 3/6] config: teach the fetch.recurseSubmodules option the 'on-demand' value

[Jens Lehmann]

To enable the user to change the default behavior of "git fetch" and "git
pull" regarding submodule recursion add the new "on-demand" value which
has just been added to the "--recurse-submodules" command line option.

Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
---
 Documentation/config.txt    |   10 +++++++---
 submodule.c                 |    2 +-
 t/t5526-fetch-submodules.sh |   28 ++++++++++++++++++++++++++++
 3 files changed, 36 insertions(+), 4 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index b4e65b8..688d4fd 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -900,9 +900,13 @@ diff.wordRegex::
 	characters are *ignorable* whitespace.

 fetch.recurseSubmodules::
-	A boolean value which changes the behavior for fetch and pull, the
-	default is to not recursively fetch populated submodules unless
-	configured otherwise.
+	This option can be either set to a boolean value or to 'on-demand'.
+	Setting it to a boolean changes the behavior of fetch and pull to
+	unconditionally recurse into submodules when set to true or to not
+	recurse at all when set to false. When set to 'on-demand' (the default
+	value), it tells fetch and pull to recurse only into those submodules
+	where new commits are recorded in the commmits fetched for the
+	superproject.

 fetch.unpackLimit::
 	If the number of objects fetched over the git native
diff --git a/submodule.c b/submodule.c
index 398616a..cccd728 100644
--- a/submodule.c
+++ b/submodule.c
@@ -71,7 +71,7 @@ int submodule_config(const char *var, const char *value, void *cb)
 	if (!prefixcmp(var, "submodule."))
 		return parse_submodule_config_option(var, value);
 	else if (!strcmp(var, "fetch.recursesubmodules")) {
-		config_fetch_recurse_submodules = git_config_bool(var, value);
+		config_fetch_recurse_submodules = parse_fetch_recurse_submodules_arg(value);
 		return 0;
 	}
 	return 0;
diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh
index 4cd723c..e6d873a 100755
--- a/t/t5526-fetch-submodules.sh
+++ b/t/t5526-fetch-submodules.sh
@@ -372,4 +372,32 @@ test_expect_success "'--recurse-submodules=on-demand' stops when no new submodul
 	test_cmp expect.err.file actual.err
 '

+test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config" '
+	(
+		cd downstream &&
+		git fetch --recurse-submodules
+	) &&
+	add_upstream_commit &&
+	git config --global fetch.recurseSubmodules false &&
+	head1=$(git rev-parse --short HEAD) &&
+	git add submodule &&
+	git commit -m "new submodule" &&
+	head2=$(git rev-parse --short HEAD) &&
+	echo "From $pwd/." > expect.err.2 &&
+	echo "   $head1..$head2  master     -> origin/master" >> expect.err.2
+	head -2 expect.err >> expect.err.2 &&
+	(
+		cd downstream &&
+		git config fetch.recurseSubmodules on-demand &&
+		git fetch >../actual.out 2>../actual.err
+	) &&
+	git config --global --unset fetch.recurseSubmodules &&
+	(
+		cd downstream &&
+		git config --unset fetch.recurseSubmodules
+	) &&
+	test_cmp expect.out.sub actual.out &&
+	test_cmp expect.err.2 actual.err
+'
+
 test_done
-- 
1.7.4.1.190.g13e20

[PATCH 4/6] Submodules: Add 'on-demand' value for the 'fetchRecurseSubmodule' option

[Jens Lehmann]

Now the behavior of fetch and pull can be configured to the recently added
'on-demand' mode separately for each submodule too.

Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
---
 Documentation/config.txt     |    2 +-
 Documentation/gitmodules.txt |    4 ++--
 submodule.c                  |    9 +++++++--
 t/t5526-fetch-submodules.sh  |   28 ++++++++++++++++++++++++++++
 4 files changed, 38 insertions(+), 5 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 688d4fd..a34b886 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -1823,7 +1823,7 @@ submodule.<name>.update::
 	linkgit:git-submodule[1] and linkgit:gitmodules[5] for details.

 submodule.<name>.fetchRecurseSubmodules::
-	This option can be used to enable/disable recursive fetching of this
+	This option can be used to control recursive fetching of this
 	submodule. It can be overridden by using the --[no-]recurse-submodules
 	command line option to "git fetch" and "git pull".
 	This setting will override that from in the linkgit:gitmodules[5]
diff --git a/Documentation/gitmodules.txt b/Documentation/gitmodules.txt
index 6897794..25daee2 100644
--- a/Documentation/gitmodules.txt
+++ b/Documentation/gitmodules.txt
@@ -45,12 +45,12 @@ submodule.<name>.update::
 	the '--merge' or '--rebase' options.

 submodule.<name>.fetchRecurseSubmodules::
-	This option can be used to enable/disable recursive fetching of this
+	This option can be used to control recursive fetching of this
 	submodule. If this option is also present in the submodules entry in
 	.git/config of the superproject, the setting there will override the
 	one found in .gitmodules.
 	Both settings can be overridden on the command line by using the
-	"--[no-]recurse-submodules" option to "git fetch" and "git pull"..
+	"--[no-]recurse-submodules" option to "git fetch" and "git pull".

 submodule.<name>.ignore::
 	Defines under what circumstances "git status" and the diff family show
diff --git a/submodule.c b/submodule.c
index cccd728..b477c3c 100644
--- a/submodule.c
+++ b/submodule.c
@@ -113,7 +113,7 @@ int parse_submodule_config_option(const char *var, const char *value)
 		if (!config)
 			config = string_list_append(&config_fetch_recurse_submodules_for_name,
 						    strbuf_detach(&submodname, NULL));
-		config->util = git_config_bool(var, value) ? (void *)1 : NULL;
+		config->util = (void *)(size_t)parse_fetch_recurse_submodules_arg(value);
 		strbuf_release(&submodname);
 	} else if ((len > 7) && !strcmp(var + len - 7, ".ignore")) {
 		if (strcmp(value, "untracked") && strcmp(value, "dirty") &&
@@ -376,8 +376,13 @@ int fetch_populated_submodules(int num_options, const char **options,
 			struct string_list_item *fetch_recurse_submodules_option;
 			fetch_recurse_submodules_option = unsorted_string_list_lookup(&config_fetch_recurse_submodules_for_name, name);
 			if (fetch_recurse_submodules_option) {
-				if (!fetch_recurse_submodules_option->util)
+				if ((size_t)fetch_recurse_submodules_option->util == RECURSE_SUBMODULES_OFF)
 					continue;
+				if ((size_t)fetch_recurse_submodules_option->util == RECURSE_SUBMODULES_ON_DEMAND) {
+					if (!unsorted_string_list_lookup(&changed_submodule_paths, ce->name))
+						continue;
+					default_argv = "on-demand";
+				}
 			} else {
 				if (config_fetch_recurse_submodules == RECURSE_SUBMODULES_OFF)
 					continue;
diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh
index e6d873a..09701aa 100755
--- a/t/t5526-fetch-submodules.sh
+++ b/t/t5526-fetch-submodules.sh
@@ -400,4 +400,32 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config
 	test_cmp expect.err.2 actual.err
 '

+test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' overrides fetch.recurseSubmodules" '
+	(
+		cd downstream &&
+		git fetch --recurse-submodules
+	) &&
+	add_upstream_commit &&
+	git config fetch.recurseSubmodules false &&
+	head1=$(git rev-parse --short HEAD) &&
+	git add submodule &&
+	git commit -m "new submodule" &&
+	head2=$(git rev-parse --short HEAD) &&
+	echo "From $pwd/." > expect.err.2 &&
+	echo "   $head1..$head2  master     -> origin/master" >> expect.err.2
+	head -2 expect.err >> expect.err.2 &&
+	(
+		cd downstream &&
+		git config submodule.submodule.fetchRecurseSubmodules on-demand &&
+		git fetch >../actual.out 2>../actual.err
+	) &&
+	git config --unset fetch.recurseSubmodules &&
+	(
+		cd downstream &&
+		git config --unset submodule.submodule.fetchRecurseSubmodules
+	) &&
+	test_cmp expect.out.sub actual.out &&
+	test_cmp expect.err.2 actual.err
+'
+
 test_done
-- 
1.7.4.1.190.g13e20

[PATCH 5/6] fetch/pull: Don't recurse into a submodule when commits are already present

[Jens Lehmann]

When looking for submodules where new commits have been recorded in the
superproject ignore those cases where the submodules commits are already
present locally. This can happen e.g. when the submodule has been rewound
to an earlier state. Then there is no need to fetch the submodule again
as the commit recorded in the newly fetched superproject commit has
already been fetched earlier into the submodule.

Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
---
 submodule.c                 |    9 ++++++++-
 t/t5526-fetch-submodules.sh |   19 +++++++++++++++++++
 2 files changed, 27 insertions(+), 1 deletions(-)

diff --git a/submodule.c b/submodule.c
index b477c3c..ddb0a29 100644
--- a/submodule.c
+++ b/submodule.c
@@ -263,6 +263,13 @@ void set_config_fetch_recurse_submodules(int value)
 	config_fetch_recurse_submodules = value;
 }

+static int is_submodule_commit_present(const char *path, unsigned char sha1[20])
+{
+	if (!add_submodule_odb(path))
+		return lookup_commit_reference(sha1) != 0;
+	return 0;
+}
+
 static void submodule_collect_changed_cb(struct diff_queue_struct *q,
 					 struct diff_options *options,
 					 void *data)
@@ -280,7 +287,7 @@ static void submodule_collect_changed_cb(struct diff_queue_struct *q,
 			 * being moved around. */
 			struct string_list_item *path;
 			path = unsorted_string_list_lookup(&changed_submodule_paths, p->two->path);
-			if (!path)
+			if (!path && !is_submodule_commit_present(p->two->path, p->two->sha1))
 				string_list_append(&changed_submodule_paths, xstrdup(p->two->path));
 		} else {
 			/* Submodule is new or was moved here */
diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh
index 09701aa..3decfae 100755
--- a/t/t5526-fetch-submodules.sh
+++ b/t/t5526-fetch-submodules.sh
@@ -428,4 +428,23 @@ test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' override
 	test_cmp expect.err.2 actual.err
 '

+test_expect_success "don't fetch submodule when newly recorded commits are already present" '
+	(
+		cd submodule &&
+		git checkout -q HEAD^^
+	) &&
+	head1=$(git rev-parse --short HEAD) &&
+	git add submodule &&
+	git commit -m "submodule rewound" &&
+	head2=$(git rev-parse --short HEAD) &&
+	echo "From $pwd/." > expect.err &&
+	echo "   $head1..$head2  master     -> origin/master" >> expect.err &&
+	(
+		cd downstream &&
+		git fetch >../actual.out 2>../actual.err
+	) &&
+	! test -s actual.out &&
+	test_cmp expect.err actual.err
+'
+
 test_done
-- 
1.7.4.1.190.g13e20

[PATCH 6/6] submodule update: Don't fetch when the submodule commit is already present

[Jens Lehmann]

If the commit to be checked out on "git submodule update" has already been
fetched in the submodule there is no need to run "git fetch" again. Since
"git fetch" recently learned recursion (and the new on-demand mode to
fetch commits recorded in the superproject is enabled by default) this
will happen pretty often, thereby making the unconditional fetch during
"git submodule update" unnecessary.

If the commit is not present in the submodule (e.g. the user disabled the
fetch on-demand mode) the fetch will be run as before.

Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
---
 git-submodule.sh            |    2 +-
 t/t7406-submodule-update.sh |   20 ++++++++++++++++++++
 2 files changed, 21 insertions(+), 1 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 8b90589..ea49760 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -465,7 +465,7 @@ cmd_update()
 			if test -z "$nofetch"
 			then
 				(clear_local_git_env; cd "$path" &&
-					git-fetch) ||
+					(git rev-parse --verify $sha1 >/dev/null || git-fetch)) ||
 				die "Unable to fetch in submodule path '$path'"
 			fi

diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
index bfb4975..ee3eec5 100755
--- a/t/t7406-submodule-update.sh
+++ b/t/t7406-submodule-update.sh
@@ -74,6 +74,26 @@ test_expect_success 'submodule update detaching the HEAD ' '
 	)
 '

+apos="'";
+test_expect_success 'submodule update does not fetch already present commits' '
+	(cd submodule &&
+	  echo line3 >> file &&
+	  git add file &&
+	  test_tick &&
+	  git commit -m "upstream line3"
+	) &&
+	(cd super/submodule &&
+	  head=$(git rev-parse --verify HEAD) &&
+	  echo "Submodule path ${apos}submodule$apos: checked out $apos$head$apos" > ../../expected &&
+	  git reset --hard HEAD~1
+	) &&
+	(cd super &&
+	  git submodule update > ../actual 2> ../actual.err
+	) &&
+	test_cmp expected actual &&
+	! test -s actual.err
+'
+
 test_expect_success 'submodule update --rebase staying on master' '
 	(cd super/submodule &&
 	  git checkout master
-- 
1.7.4.1.190.g13e20

Re: [PATCH 1/6] fetch/pull: recurse into submodules when necessary

[Junio C Hamano]

Jens Lehmann <Jens.Lehmann@web.de> writes:

> diff --git a/submodule.c b/submodule.c
> index 6f1c107..c8c3a99 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -152,6 +153,20 @@ void handle_ignore_submodules_arg(struct diff_options *diffopt,
> ...
> +int parse_fetch_recurse_submodules_arg(const char *arg)
> +{
> +	switch (git_config_maybe_bool("", arg)) {

It's a bit unusual to see "" as the variable name; doesn't config-maybe-bool
barf when arg is not what it likes, with the name as part of its message?

> @@ -248,11 +263,73 @@ void set_config_fetch_recurse_submodules(int value)
> ...
> +static void submodule_collect_changed_cb(struct diff_queue_struct *q,
> +					 struct diff_options *options,
> +					 void *data)
> +{
> +	int i;
> +	for (i = 0; i < q->nr; i++) {
> +		struct diff_filepair *p = q->queue[i];
> +		if (!S_ISGITLINK(p->two->mode))
> +			continue;
> +
> +		if (S_ISGITLINK(p->one->mode)) {
> +			/* NEEDSWORK: We should honor the name configured in
> +			 * the .gitmodules file of the commit we are examining
> +			 * here to be able to correctly follow submodules
> +			 * being moved around. */
> +			struct string_list_item *path;
> +			path = unsorted_string_list_lookup(&changed_submodule_paths, p->two->path);
> +			if (!path)
> +				string_list_append(&changed_submodule_paths, xstrdup(p->two->path));

I wondered why there is no mention of "data" in the implementation of this
function; changed_submodule_paths global is used instead here.

I do not see anywhere the global string_list is initialized, freed, nor
re-initialized for reuse.  Are you guaranteeing that the caller only calls
the check_for_new_submodule_commits() function once, and if so how?  The
function is called from update_local_ref() in builtin/fetch.c, which in
turn gets called number of times during a fetch.  IOW, does the code do
the right thing when you are fetching more than one refs?

Re: [PATCH 1/6] fetch/pull: recurse into submodules when necessary

[Jonathan Nieder]

Jens Lehmann wrote:

> To be able to access all commits of populated submodules referenced by the
> superproject it is sufficient to only then let "git fetch" recurse into a
> submodule when the new commits fetched in the superproject record new
> commits for it.

Exciting stuff.  This would use the default refspec rather than fetching
the bare minimum of commits in submodules, right?

> There is currently no infrastructure for storing deleted and new
> submodules in the .git directory of the superproject. Thats why fetch and
> pull for now only fetch submodules that are already checked out and are
> not renamed.

Ah.

> --- a/Documentation/fetch-options.txt
> +++ b/Documentation/fetch-options.txt
> @@ -73,6 +73,14 @@ ifndef::git-pull[]
>  	Prepend <path> to paths printed in informative messages
>  	such as "Fetching submodule foo".  This option is used
>  	internally when recursing over submodules.
> +
> +--submodule-default=[yes|on-demand]::
> +	This option is used internally to set the submodule recursion default
> +	to either a boolean configuration value representing "true" (for
> +	unconditonal recursion) or to "on-demand" (when only those submodules
> +	should be fetched of which new commits have been fetched in its
> +	superproject).
> +	Using the "--[no-]recurse-submodules" option ignores this setting.

Could this be combined with the --recurse-submodules, with a "last instance
of the option wins" rule?  Something like:

 --recurse-submodules[=(yes | no | changed)]::
 --no-recurse-submodules::

> +++ b/submodule.c
[...]
> @@ -248,11 +263,73 @@ void set_config_fetch_recurse_submodules(int value)
[...]
> +static void submodule_collect_changed_cb(struct diff_queue_struct *q,
[...]
> +		if (S_ISGITLINK(p->one->mode)) {
> +			/* NEEDSWORK: We should honor the name configured in
> +			 * the .gitmodules file of the commit we are examining
> +			 * here to be able to correctly follow submodules
> +			 * being moved around. */

Hmm, a sort of variant on rename detection.  Does "git submodule update" /
"git checkout --recurse-submodules" use .gitmodules to move pre-populated
subrepositories?

> +			/* Submodule is new or was moved here */
> +			/* NEEDSWORK: When the .git directories of submodules
> +			 * live inside the superprojects .git directory some
> +			 * day we should fetch new submodules directly into
> +			 * that location too when config or options request
> +			 * that so they can be checked out from there. */
> +			continue;

Maybe this can be mentioned in a BUGS section on the git-fetch(1)
manpage to give readers a warning and clue about the intended
meaning of --recurse-submodules?

I'm afraid a certain kind of person might get used to the "don't fetch
new submodules" behavior (e.g., if upstream is including unneeded
convenience copies of libraries right and left in addition to some
useful submodules).

> +void check_for_new_submodule_commits(unsigned char new_sha1[20])
> +{
> +	struct rev_info rev;
> +	struct commit *commit;
> +	int argc = 5;
> +	const char *argv[] = {NULL, NULL, "--not", "--branches", "--remotes", NULL};

Nit: maybe

	const char *argv[] = ...;
	int argc = ARRAY_SIZE(argv) - 1;

?

> +
> +	init_revisions(&rev, NULL);
> +	argv[1] = xstrdup(sha1_to_hex(new_sha1));

Maybe:

	char *new_rev;
	...
	argv[1] = new_rev = xstrdup(...);
	...
	free(new_rev);

> +	setup_revisions(argc, argv, &rev, NULL);
> +	if (prepare_revision_walk(&rev))
> +		die("revision walk setup failed");
> +

Maybe a comment so the reader doesn't have to delve deeper?

	/*
	 * Collect checked out submodules that have changed upstream
	 * in "changed_submodule_paths".
	 */

> +	while ((commit = get_revision(&rev))) {
[...]
> +++ b/t/t5526-fetch-submodules.sh
> @@ -192,4 +192,113 @@ test_expect_success "--no-recurse-submodules overrides config setting" '
[...]
> +	echo "Fetching submodule submodule" > expect.out.sub &&
> +	echo "From $pwd/." > expect.err.sub &&
> +	echo "   $head1..$head2  master     -> origin/master" >> expect.err.sub

I wonder if we should be testing the output in this much detail.

Wouldn't it be nicer to check the remote-tracking refs to make sure
the lasting effects were correct, rather than the detailed output
format?

So: aside from the option name, nothing but nits.  Thanks; that was
fun.

Re: [PATCH 2/6] fetch/pull: Add the 'on-demand' value to the --recurse-submodules option

[Jonathan Nieder]

Jens Lehmann wrote:

>             As fetch and pull now by default just fetch those submodules
> for which new commits have been fetched in the superproject, a command
> line option to enforce that behavior is needed to be able to override
> configuration settings.

Probably this should go first in the series (the usual procedure:
first command line for easy testing, then configuration for routine
use, then defaults).

Aside from that, it looks sane from skimming this over.  I think I
prefer the name "changed" over "on-demand" since it is a little more
obvious what it will do.  Neither name captures that this only affects
checked-out submodules, unfortunately.

Re: [PATCH 2/6] fetch/pull: Add the 'on-demand' value to the --recurse-submodules option

[Jonathan Nieder]

Jonathan Nieder wrote:
> Jens Lehmann wrote:

>>             As fetch and pull now by default just fetch those submodules
>> for which new commits have been fetched in the superproject, a command
>> line option to enforce that behavior is needed to be able to override
>> configuration settings.
>
> Probably this should go first in the series (the usual procedure:
> first command line for easy testing, then configuration for routine
> use, then defaults).

Ah, sorry for the nonsense.  The series already follows that order.

Re: [PATCH 4/6] Submodules: Add 'on-demand' value for the 'fetchRecurseSubmodule' option

[Junio C Hamano]

Jens Lehmann <Jens.Lehmann@web.de> writes:

> diff --git a/submodule.c b/submodule.c
> index cccd728..b477c3c 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -113,7 +113,7 @@ int parse_submodule_config_option(const char *var, const char *value)
>  		if (!config)
>  			config = string_list_append(&config_fetch_recurse_submodules_for_name,
>  						    strbuf_detach(&submodname, NULL));
> -		config->util = git_config_bool(var, value) ? (void *)1 : NULL;
> +		config->util = (void *)(size_t)parse_fetch_recurse_submodules_arg(value);

What is this double-cast about?

> @@ -376,8 +376,13 @@ int fetch_populated_submodules(int num_options, const char **options,
> ...
> +				if ((size_t)fetch_recurse_submodules_option->util == RECURSE_SUBMODULES_OFF)
>  					continue;
> +				if ((size_t)fetch_recurse_submodules_option->util == RECURSE_SUBMODULES_ON_DEMAND) {

Likewise here; size_t feels a strange type to cast to in this comparison
between (void *) and an enum, no?

Re: [PATCH 5/6] fetch/pull: Don't recurse into a submodule when commits are already present

[Junio C Hamano]

Jens Lehmann <Jens.Lehmann@web.de> writes:

> diff --git a/submodule.c b/submodule.c
> index b477c3c..ddb0a29 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -263,6 +263,13 @@ void set_config_fetch_recurse_submodules(int value)
>  	config_fetch_recurse_submodules = value;
>  }
>
> +static int is_submodule_commit_present(const char *path, unsigned char sha1[20])
> +{
> +	if (!add_submodule_odb(path))
> +		return lookup_commit_reference(sha1) != 0;
> +	return 0;
> +}

Don't you need to prove the usual "reachabile from the refs" here, instead
of just the presense of a commit object?

If not, why not?

Re: [PATCH 0/6] Teach fetch/pull the on-demand mode and make it the default

[Jonathan Nieder]

Jens Lehmann wrote:

> *) The fetch is only done when the recorded submodule commit isn't
>    already present.

I like this part a lot.

> I tend to think that this is suited for 1.7.5 but don't have any
> objections against holding it back until 1.8.0 either. What do
> others think?

I see no backward-compatibility to wait for this, but I would be more
included to trust people using "git submodule update" heavily than I
do.  The "submodule update" change could cause the following to break.
Would that be disruptive?

	cd submodule
	git fetch --no-recurse-submodules
	...

	cd ..
	bin/script-to-update-submodules-that-calls-submodule-update

Thanks a lot for working on this.
Jonathan

Re: [PATCH 6/6] submodule update: Don't fetch when the submodule commit is already present

[Junio C Hamano]

Jens Lehmann <Jens.Lehmann@web.de> writes:

> diff --git a/git-submodule.sh b/git-submodule.sh
> index 8b90589..ea49760 100755
> --- a/git-submodule.sh
> +++ b/git-submodule.sh
> @@ -465,7 +465,7 @@ cmd_update()
>  			if test -z "$nofetch"
>  			then
>  				(clear_local_git_env; cd "$path" &&
> -					git-fetch) ||
> +					(git rev-parse --verify $sha1 >/dev/null || git-fetch)) ||

Exactly the same comment as the one for [5/6] applies here.

Re: [PATCH 1/6] fetch/pull: recurse into submodules when necessary

[Jens Lehmann]

Am 23.02.2011 23:56, schrieb Junio C Hamano:
> Jens Lehmann <Jens.Lehmann@web.de> writes:
> 
>> diff --git a/submodule.c b/submodule.c
>> index 6f1c107..c8c3a99 100644
>> --- a/submodule.c
>> +++ b/submodule.c
>> @@ -152,6 +153,20 @@ void handle_ignore_submodules_arg(struct diff_options *diffopt,
>> ...
>> +int parse_fetch_recurse_submodules_arg(const char *arg)
>> +{
>> +	switch (git_config_maybe_bool("", arg)) {
> 
> It's a bit unusual to see "" as the variable name; doesn't config-maybe-bool
> barf when arg is not what it likes, with the name as part of its message?

git_config_*maybe*_bool() itself calls git_config_maybe_bool_text() and
git_parse_long() which all don't die() on anything (while git_config_bool()
can do that in git_config_int() via git_config_bool_or_int()). But you are
right, using something more descriptive than "" is much better here.

>> @@ -248,11 +263,73 @@ void set_config_fetch_recurse_submodules(int value)
>> ...
>> +static void submodule_collect_changed_cb(struct diff_queue_struct *q,
>> +					 struct diff_options *options,
>> +					 void *data)
>> +{
>> +	int i;
>> +	for (i = 0; i < q->nr; i++) {
>> +		struct diff_filepair *p = q->queue[i];
>> +		if (!S_ISGITLINK(p->two->mode))
>> +			continue;
>> +
>> +		if (S_ISGITLINK(p->one->mode)) {
>> +			/* NEEDSWORK: We should honor the name configured in
>> +			 * the .gitmodules file of the commit we are examining
>> +			 * here to be able to correctly follow submodules
>> +			 * being moved around. */
>> +			struct string_list_item *path;
>> +			path = unsorted_string_list_lookup(&changed_submodule_paths, p->two->path);
>> +			if (!path)
>> +				string_list_append(&changed_submodule_paths, xstrdup(p->two->path));
> 
> I wondered why there is no mention of "data" in the implementation of this
> function; changed_submodule_paths global is used instead here.
> 
> I do not see anywhere the global string_list is initialized, freed, nor
> re-initialized for reuse.  Are you guaranteeing that the caller only calls
> the check_for_new_submodule_commits() function once, and if so how?  The
> function is called from update_local_ref() in builtin/fetch.c, which in
> turn gets called number of times during a fetch.  IOW, does the code do
> the right thing when you are fetching more than one refs?

I assume that a string_list initialized with 0 is initialized properly.
The idea is to let check_for_new_submodule_commits() be called as often
as needed and for each ref to collect all submodule changes recorded in
any ref and afterwards only once call fetch_populated_submodules() to
collect all submodules touched by any commits on any ref. But maybe
fetch_populated_submodules() should empty the string_list it just
worked through?

Re: [PATCH 1/6] fetch/pull: recurse into submodules when necessary

[Jens Lehmann]

Am 24.02.2011 00:07, schrieb Jonathan Nieder:
> Jens Lehmann wrote:
> 
>> To be able to access all commits of populated submodules referenced by the
>> superproject it is sufficient to only then let "git fetch" recurse into a
>> submodule when the new commits fetched in the superproject record new
>> commits for it.
> 
> Exciting stuff.  This would use the default refspec rather than fetching
> the bare minimum of commits in submodules, right?

Yup, AFAIK I have no means right now to tell "git fetch" to only fetch
certain commits. But we were talking at the GitTogether that this might
be useful and with a bit of tweaking the code would support that too.

>> --- a/Documentation/fetch-options.txt
>> +++ b/Documentation/fetch-options.txt
>> @@ -73,6 +73,14 @@ ifndef::git-pull[]
>>  	Prepend <path> to paths printed in informative messages
>>  	such as "Fetching submodule foo".  This option is used
>>  	internally when recursing over submodules.
>> +
>> +--submodule-default=[yes|on-demand]::
>> +	This option is used internally to set the submodule recursion default
>> +	to either a boolean configuration value representing "true" (for
>> +	unconditonal recursion) or to "on-demand" (when only those submodules
>> +	should be fetched of which new commits have been fetched in its
>> +	superproject).
>> +	Using the "--[no-]recurse-submodules" option ignores this setting.
> 
> Could this be combined with the --recurse-submodules, with a "last instance
> of the option wins" rule?  Something like:
> 
>  --recurse-submodules[=(yes | no | changed)]::
>  --no-recurse-submodules::

Nope, as this only sets the default. "--recurse-submodules" overrides
anything configured, which is not what we want here. This option should
only set the default.

>> +++ b/submodule.c
> [...]
>> @@ -248,11 +263,73 @@ void set_config_fetch_recurse_submodules(int value)
> [...]
>> +static void submodule_collect_changed_cb(struct diff_queue_struct *q,
> [...]
>> +		if (S_ISGITLINK(p->one->mode)) {
>> +			/* NEEDSWORK: We should honor the name configured in
>> +			 * the .gitmodules file of the commit we are examining
>> +			 * here to be able to correctly follow submodules
>> +			 * being moved around. */
> 
> Hmm, a sort of variant on rename detection.  Does "git submodule update" /
> "git checkout --recurse-submodules" use .gitmodules to move pre-populated
> subrepositories?

Not yet ;-)

>> +			/* Submodule is new or was moved here */
>> +			/* NEEDSWORK: When the .git directories of submodules
>> +			 * live inside the superprojects .git directory some
>> +			 * day we should fetch new submodules directly into
>> +			 * that location too when config or options request
>> +			 * that so they can be checked out from there. */
>> +			continue;
> 
> Maybe this can be mentioned in a BUGS section on the git-fetch(1)
> manpage to give readers a warning and clue about the intended
> meaning of --recurse-submodules?
> 
> I'm afraid a certain kind of person might get used to the "don't fetch
> new submodules" behavior (e.g., if upstream is including unneeded
> convenience copies of libraries right and left in addition to some
> useful submodules).

I'm not sure I understand what you mean by this, right now this can
only work for populated submodules. I hope this will change soon, but
I'm not quite there yet ;-)

>> +void check_for_new_submodule_commits(unsigned char new_sha1[20])
>> +{
>> +	struct rev_info rev;
>> +	struct commit *commit;
>> +	int argc = 5;
>> +	const char *argv[] = {NULL, NULL, "--not", "--branches", "--remotes", NULL};
> 
> Nit: maybe
> 
> 	const char *argv[] = ...;
> 	int argc = ARRAY_SIZE(argv) - 1;
> 
> ?

Fine by me!

>> +
>> +	init_revisions(&rev, NULL);
>> +	argv[1] = xstrdup(sha1_to_hex(new_sha1));
> 
> Maybe:
> 
> 	char *new_rev;
> 	...
> 	argv[1] = new_rev = xstrdup(...);
> 	...
> 	free(new_rev);

I'm not sure I get what the extra variable gains us ...

>> +	setup_revisions(argc, argv, &rev, NULL);
>> +	if (prepare_revision_walk(&rev))
>> +		die("revision walk setup failed");
>> +
> 
> Maybe a comment so the reader doesn't have to delve deeper?

?

> 	/*
> 	 * Collect checked out submodules that have changed upstream
> 	 * in "changed_submodule_paths".
> 	 */
> 
>> +	while ((commit = get_revision(&rev))) {
> [...]
>> +++ b/t/t5526-fetch-submodules.sh
>> @@ -192,4 +192,113 @@ test_expect_success "--no-recurse-submodules overrides config setting" '
> [...]
>> +	echo "Fetching submodule submodule" > expect.out.sub &&
>> +	echo "From $pwd/." > expect.err.sub &&
>> +	echo "   $head1..$head2  master     -> origin/master" >> expect.err.sub
> 
> I wonder if we should be testing the output in this much detail.
> 
> Wouldn't it be nicer to check the remote-tracking refs to make sure
> the lasting effects were correct, rather than the detailed output
> format?

Yes, that makes sense!

> So: aside from the option name, nothing but nits.  Thanks; that was
> fun.

Thanks!

Re: [PATCH 0/6] Teach fetch/pull the on-demand mode and make it the default

[Jens Lehmann]

Am 24.02.2011 00:21, schrieb Jonathan Nieder:
> Jens Lehmann wrote:
> 
>> *) The fetch is only done when the recorded submodule commit isn't
>>    already present.
> 
> I like this part a lot.

Glad to hear ;-)

>> I tend to think that this is suited for 1.7.5 but don't have any
>> objections against holding it back until 1.8.0 either. What do
>> others think?
> 
> I see no backward-compatibility to wait for this, but I would be more
> included to trust people using "git submodule update" heavily than I
> do.

Yeah, I would appreciate some feedback here too.

> The "submodule update" change could cause the following to break.
> Would that be disruptive?
> 
> 	cd submodule
> 	git fetch --no-recurse-submodules
> 	...
> 
> 	cd ..
> 	bin/script-to-update-submodules-that-calls-submodule-update

But then "git submodule update" would notice that the commit isn't
present and do a "git fetch" itself, no?

Re: [PATCH 5/6] fetch/pull: Don't recurse into a submodule when commits are already present

[Jens Lehmann]

Am 24.02.2011 00:21, schrieb Junio C Hamano:
> Jens Lehmann <Jens.Lehmann@web.de> writes:
> 
>> diff --git a/submodule.c b/submodule.c
>> index b477c3c..ddb0a29 100644
>> --- a/submodule.c
>> +++ b/submodule.c
>> @@ -263,6 +263,13 @@ void set_config_fetch_recurse_submodules(int value)
>>  	config_fetch_recurse_submodules = value;
>>  }
>>
>> +static int is_submodule_commit_present(const char *path, unsigned char sha1[20])
>> +{
>> +	if (!add_submodule_odb(path))
>> +		return lookup_commit_reference(sha1) != 0;
>> +	return 0;
>> +}
> 
> Don't you need to prove the usual "reachabile from the refs" here, instead
> of just the presense of a commit object?

Looks like I would. Could you please enlighten me?

Re: [PATCH 1/6] fetch/pull: recurse into submodules when necessary

[Jonathan Nieder]

Jens Lehmann wrote:
> Am 24.02.2011 00:07, schrieb Jonathan Nieder:

>> Could this be combined with the --recurse-submodules, with a "last instance
>> of the option wins" rule?  Something like:
>> 
>>  --recurse-submodules[=(yes | no | changed)]::
>>  --no-recurse-submodules::
>
> Nope, as this only sets the default. "--recurse-submodules" overrides
> anything configured, which is not what we want here. This option should
> only set the default.

Ah.  So --recurse-submodules-default means "like --recurse-submodules,
but with lower precedence than the configuration".  Sensible.  (Maybe
it could be documented in --help-all that way?)

>>> +			/* Submodule is new or was moved here */
>>> +			/* NEEDSWORK: When the .git directories of submodules
>>> +			 * live inside the superprojects .git directory some
>>> +			 * day we should fetch new submodules directly into
>>> +			 * that location too when config or options request
>>> +			 * that so they can be checked out from there. */
>>> +			continue;
>> 
>> Maybe this can be mentioned in a BUGS section on the git-fetch(1)
>> manpage to give readers a warning and clue about the intended
>> meaning of --recurse-submodules?
[...]
> I'm not sure I understand what you mean by this, right now this can
> only work for populated submodules. I hope this will change soon, but
> I'm not quite there yet ;-)

What I mean is the following: to make life easier for people and
scripts using --recurse-submodules today, it might be nice to
document how stable or unstable its meaning is.

In this case, there is a plan to make --recurse-submodules=on-demand
do more in the future than it does now;

 - a note in BUGS could explain that --recurse-submodules's current
   behavior is considered an infelicity and likely to change;

 - unfortunately not all users will necessarily see it that way (c.f.
   aforementioned use case), so it might be better to plan on yet
   another choice in the list of options provided by
   --recurse-submodules.

>> Maybe:
>>
>> 	char *new_rev;
>> 	...
>> 	argv[1] = new_rev = xstrdup(...);
>> 	...
>> 	free(new_rev);
>
> I'm not sure I get what the extra variable gains us ...

A reminder to free that memory in all code paths exiting the function.
But it might not be worth the noise.

>> Maybe a comment so the reader doesn't have to delve deeper?
>
> ?

Sorry for the confusion.  That suggestion refers to the loop after it
rather than what comes before it.

>> 	/*
>> 	 * Collect checked out submodules that have changed upstream
>> 	 * in "changed_submodule_paths".
>> 	 */
>> 
>>> +	while ((commit = get_revision(&rev))) {
[...]
> Thanks!

Thanks for deciphering the gibberish I sent. :)
Jonathan

Re: [PATCH 1/6] fetch/pull: recurse into submodules when necessary

[Jonathan Nieder]

Jens Lehmann wrote:

> But maybe
> fetch_populated_submodules() should empty the string_list it just
> worked through?

No, I don't think so.  There might be cases where you want to fetch
and then check out the changed submodules some day.

What might be nice is for cmd_fetch to clear the list before it
returns.  That way, if this functionality ever gets lib-ified then
there will be that reminder about how to reset the state.  And
valgrind can be a little quieter. :)

Re: [PATCH 5/6] fetch/pull: Don't recurse into a submodule when commits are already present

[Jens Lehmann]

Am 24.02.2011 00:50, schrieb Jens Lehmann:
> Am 24.02.2011 00:21, schrieb Junio C Hamano:
>> Jens Lehmann <Jens.Lehmann@web.de> writes:
>>
>>> diff --git a/submodule.c b/submodule.c
>>> index b477c3c..ddb0a29 100644
>>> --- a/submodule.c
>>> +++ b/submodule.c
>>> @@ -263,6 +263,13 @@ void set_config_fetch_recurse_submodules(int value)
>>>  	config_fetch_recurse_submodules = value;
>>>  }
>>>
>>> +static int is_submodule_commit_present(const char *path, unsigned char sha1[20])
>>> +{
>>> +	if (!add_submodule_odb(path))
>>> +		return lookup_commit_reference(sha1) != 0;
>>> +	return 0;
>>> +}
>>
>> Don't you need to prove the usual "reachabile from the refs" here, instead
>> of just the presense of a commit object?
> 
> Looks like I would. Could you please enlighten me?

Of course you are right. Will change that.

Re: [PATCH 4/6] Submodules: Add 'on-demand' value for the 'fetchRecurseSubmodule' option

[Jens Lehmann]

Am 24.02.2011 00:16, schrieb Junio C Hamano:
> Jens Lehmann <Jens.Lehmann@web.de> writes:
> 
>> diff --git a/submodule.c b/submodule.c
>> index cccd728..b477c3c 100644
>> --- a/submodule.c
>> +++ b/submodule.c
>> @@ -113,7 +113,7 @@ int parse_submodule_config_option(const char *var, const char *value)
>>  		if (!config)
>>  			config = string_list_append(&config_fetch_recurse_submodules_for_name,
>>  						    strbuf_detach(&submodname, NULL));
>> -		config->util = git_config_bool(var, value) ? (void *)1 : NULL;
>> +		config->util = (void *)(size_t)parse_fetch_recurse_submodules_arg(value);
> 
> What is this double-cast about?

64bit gcc warns when I drop either of them because a 32bit integer
is assigned to a 64bit wide pointer here.

>> @@ -376,8 +376,13 @@ int fetch_populated_submodules(int num_options, const char **options,
>> ...
>> +				if ((size_t)fetch_recurse_submodules_option->util == RECURSE_SUBMODULES_OFF)
>>  					continue;
>> +				if ((size_t)fetch_recurse_submodules_option->util == RECURSE_SUBMODULES_ON_DEMAND) {
> 
> Likewise here; size_t feels a strange type to cast to in this comparison
> between (void *) and an enum, no?

I get a warning here if I drop the second cast. gcc doesn't warn if
I drop the first one, but that is just because the enum value happens
to be 0 there. So I added that cast there too to be on the safe side
in case the value changes in the future and to be consistent to other
readers of this code.